2024-09-15T21:13:04.260 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T21:13:04.272 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T21:13:04.353 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905692 branch: main description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905692' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 workunit: branch: main sha1: 7498493279982dfe87f00616198a5967475f6169 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2855 sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 sleep_before_teardown: 0 subset: 17936/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7498493279982dfe87f00616198a5967475f6169 targets: smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNKhTIIh81K43sRWK1GjD4ihmUOJWgI1gxjXuIVfJU5zfnMzKd9AXvns3hO3XrI2Cv5Ue337B+2GkmJzJ637ktY= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_20:00:16 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T21:13:04.354 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa; will attempt to use it 2024-09-15T21:13:04.354 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks 2024-09-15T21:13:04.354 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T21:13:04.356 INFO:teuthology.task.internal:Checking packages... 2024-09-15T21:13:04.377 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'fc3e0bff7468fd55478adbae7f75508b749124b6' 2024-09-15T21:13:04.378 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T21:13:04.378 INFO:teuthology.packaging:ref: None 2024-09-15T21:13:04.378 INFO:teuthology.packaging:tag: None 2024-09-15T21:13:04.378 INFO:teuthology.packaging:branch: main 2024-09-15T21:13:04.378 INFO:teuthology.packaging:sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T21:13:04.378 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-09-15T21:13:04.597 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T21:13:04.598 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T21:13:04.606 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T21:13:04.606 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T21:13:04.643 INFO:teuthology.task.internal:Saving configuration 2024-09-15T21:13:04.652 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T21:13:04.675 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T21:13:04.700 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905692', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 21:07:30.482425', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICNQiWcO9oyrWjHe5/YxxC/f6naSGamxZhjzcpgdaFag'} 2024-09-15T21:13:04.701 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T21:13:04.707 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-15T21:13:04.707 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T21:13:04.755 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f44ee9cb6d0>, signals=[15]) 2024-09-15T21:13:04.755 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T21:13:04.763 INFO:teuthology.task.internal:Opening connections... 2024-09-15T21:13:04.763 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T21:13:04.764 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:13:04.846 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T21:13:04.852 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-09-15T21:13:04.945 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-09-15T21:13:04.946 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-09-15T21:13:04.990 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-15T21:13:04.991 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Ubuntu" 2024-09-15T21:13:04.991 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="22.04" 2024-09-15T21:13:04.991 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-15T21:13:04.991 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_CODENAME=jammy 2024-09-15T21:13:04.991 INFO:teuthology.orchestra.run.smithi104.stdout:ID=ubuntu 2024-09-15T21:13:04.991 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE=debian 2024-09-15T21:13:04.991 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-15T21:13:04.991 INFO:teuthology.orchestra.run.smithi104.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-15T21:13:04.991 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-15T21:13:04.991 INFO:teuthology.orchestra.run.smithi104.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-15T21:13:04.991 INFO:teuthology.orchestra.run.smithi104.stdout:UBUNTU_CODENAME=jammy 2024-09-15T21:13:04.992 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-09-15T21:13:05.017 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T21:13:05.028 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T21:13:05.078 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T21:13:05.078 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-09-15T21:13:05.083 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T21:13:05.098 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T21:13:05.098 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-09-15T21:13:05.132 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T21:13:05.169 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T21:13:05.191 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T21:13:05.191 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T21:13:05.192 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T21:13:05.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:13:05.197 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-09-15T21:13:05.243 INFO:teuthology.orchestra.run.smithi104.stdout:5.15.0-56-generic 2024-09-15T21:13:05.243 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.15.0-56-generic 2024-09-15T21:13:05.243 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get clean 2024-09-15T21:13:05.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-09-15T21:13:05.512 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-15T21:13:05.616 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T21:13:05.698 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-15T21:13:05.730 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-15T21:13:05.797 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-15T21:13:05.837 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-15T21:13:05.840 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-15T21:13:05.840 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-15T21:13:05.862 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-15T21:13:05.863 INFO:teuthology.orchestra.run.smithi104.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-15T21:13:05.866 INFO:teuthology.orchestra.run.smithi104.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-15T21:13:05.866 INFO:teuthology.orchestra.run.smithi104.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-15T21:13:05.871 INFO:teuthology.orchestra.run.smithi104.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-15T21:13:05.877 INFO:teuthology.orchestra.run.smithi104.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-15T21:13:05.879 INFO:teuthology.orchestra.run.smithi104.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-15T21:13:05.879 INFO:teuthology.orchestra.run.smithi104.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-15T21:13:05.879 INFO:teuthology.orchestra.run.smithi104.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-15T21:13:05.909 INFO:teuthology.orchestra.run.smithi104.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-15T21:13:05.909 INFO:teuthology.orchestra.run.smithi104.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-15T21:13:06.053 INFO:teuthology.orchestra.run.smithi104.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-15T21:13:06.192 INFO:teuthology.orchestra.run.smithi104.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-15T21:13:06.386 INFO:teuthology.orchestra.run.smithi104.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-15T21:13:06.545 INFO:teuthology.orchestra.run.smithi104.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-15T21:13:06.559 INFO:teuthology.orchestra.run.smithi104.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-15T21:13:06.559 INFO:teuthology.orchestra.run.smithi104.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-15T21:13:06.561 INFO:teuthology.orchestra.run.smithi104.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-15T21:13:06.630 INFO:teuthology.orchestra.run.smithi104.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-15T21:13:06.644 INFO:teuthology.orchestra.run.smithi104.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-15T21:13:06.645 INFO:teuthology.orchestra.run.smithi104.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-15T21:13:06.678 INFO:teuthology.orchestra.run.smithi104.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-15T21:13:06.697 INFO:teuthology.orchestra.run.smithi104.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-15T21:13:06.703 INFO:teuthology.orchestra.run.smithi104.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-15T21:13:06.709 INFO:teuthology.orchestra.run.smithi104.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-15T21:13:06.709 INFO:teuthology.orchestra.run.smithi104.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-15T21:13:06.711 INFO:teuthology.orchestra.run.smithi104.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-15T21:13:06.785 INFO:teuthology.orchestra.run.smithi104.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-15T21:13:06.786 INFO:teuthology.orchestra.run.smithi104.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-15T21:13:06.787 INFO:teuthology.orchestra.run.smithi104.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-15T21:13:06.789 INFO:teuthology.orchestra.run.smithi104.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-15T21:13:06.789 INFO:teuthology.orchestra.run.smithi104.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-15T21:13:06.790 INFO:teuthology.orchestra.run.smithi104.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-15T21:13:06.791 INFO:teuthology.orchestra.run.smithi104.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-15T21:13:06.872 INFO:teuthology.orchestra.run.smithi104.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-15T21:13:06.872 INFO:teuthology.orchestra.run.smithi104.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-15T21:13:12.284 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 16.0 MB in 2s (7,510 kB/s) 2024-09-15T21:13:13.726 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-15T21:13:13.744 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T21:13:13.841 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-15T21:13:14.025 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-09-15T21:13:14.025 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-09-15T21:13:14.193 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T21:13:14.193 INFO:teuthology.orchestra.run.smithi104.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T21:13:14.193 INFO:teuthology.orchestra.run.smithi104.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T21:13:14.193 INFO:teuthology.orchestra.run.smithi104.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T21:13:14.193 INFO:teuthology.orchestra.run.smithi104.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-15T21:13:14.193 INFO:teuthology.orchestra.run.smithi104.stdout: python2.7-minimal 2024-09-15T21:13:14.194 INFO:teuthology.orchestra.run.smithi104.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T21:13:14.195 INFO:teuthology.orchestra.run.smithi104.stdout:The following additional packages will be installed: 2024-09-15T21:13:14.195 INFO:teuthology.orchestra.run.smithi104.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-15T21:13:14.195 INFO:teuthology.orchestra.run.smithi104.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-15T21:13:14.196 INFO:teuthology.orchestra.run.smithi104.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-15T21:13:14.197 INFO:teuthology.orchestra.run.smithi104.stdout:Suggested packages: 2024-09-15T21:13:14.197 INFO:teuthology.orchestra.run.smithi104.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-15T21:13:14.197 INFO:teuthology.orchestra.run.smithi104.stdout:Recommended packages: 2024-09-15T21:13:14.197 INFO:teuthology.orchestra.run.smithi104.stdout: thermald 2024-09-15T21:13:14.243 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2024-09-15T21:13:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-15T21:13:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-15T21:13:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-15T21:13:14.244 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages will be upgraded: 2024-09-15T21:13:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-15T21:13:14.443 INFO:teuthology.orchestra.run.smithi104.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-15T21:13:14.443 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 113 MB of archives. 2024-09-15T21:13:14.443 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-15T21:13:14.443 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-15T21:13:15.937 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-15T21:13:16.288 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-15T21:13:18.519 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-15T21:13:18.519 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-15T21:13:18.519 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-15T21:13:18.855 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-15T21:13:18.936 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-15T21:13:19.333 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 113 MB in 5s (24.2 MB/s) 2024-09-15T21:13:19.506 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-15T21:13:21.442 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-15T21:13:21.446 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T21:13:21.603 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T21:13:23.662 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-15T21:13:23.683 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T21:13:23.763 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T21:13:24.232 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-15T21:13:24.252 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T21:13:24.290 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T21:13:30.773 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T21:13:30.900 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T21:13:31.224 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T21:13:31.362 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T21:13:31.665 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-15T21:13:31.687 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-15T21:13:31.731 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-15T21:13:37.187 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-15T21:13:37.223 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T21:13:37.261 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T21:13:40.395 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T21:13:40.520 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T21:13:40.861 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-15T21:13:40.991 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T21:13:41.190 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-15T21:13:42.020 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T21:13:43.352 INFO:teuthology.orchestra.run.smithi104.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-15T21:13:43.352 INFO:teuthology.orchestra.run.smithi104.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-15T21:13:43.352 INFO:teuthology.orchestra.run.smithi104.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-15T21:13:43.353 INFO:teuthology.orchestra.run.smithi104.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-15T21:13:43.541 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T21:13:44.916 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T21:13:46.260 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-15T21:13:46.369 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-15T21:13:46.495 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T21:13:46.555 INFO:teuthology.orchestra.run.smithi104.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-15T21:13:46.555 INFO:teuthology.orchestra.run.smithi104.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-15T21:14:00.275 INFO:teuthology.orchestra.run.smithi104.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-15T21:14:00.275 INFO:teuthology.orchestra.run.smithi104.stdout:Sourcing file `/etc/default/grub' 2024-09-15T21:14:00.286 INFO:teuthology.orchestra.run.smithi104.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T21:14:00.309 INFO:teuthology.orchestra.run.smithi104.stdout:Generating grub configuration file ... 2024-09-15T21:14:00.568 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-15T21:14:00.587 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-15T21:14:00.888 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-15T21:14:00.894 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-15T21:14:00.963 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-15T21:14:00.969 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-15T21:14:01.141 INFO:teuthology.orchestra.run.smithi104.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-15T21:14:01.141 INFO:teuthology.orchestra.run.smithi104.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-15T21:14:01.141 INFO:teuthology.orchestra.run.smithi104.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-15T21:14:01.183 INFO:teuthology.orchestra.run.smithi104.stdout:done 2024-09-15T21:14:01.698 DEBUG:teuthology.orchestra.run.smithi104:> dpkg -s linux-image-generic 2024-09-15T21:14:01.720 INFO:teuthology.orchestra.run.smithi104.stdout:Package: linux-image-generic 2024-09-15T21:14:01.720 INFO:teuthology.orchestra.run.smithi104.stdout:Status: install ok installed 2024-09-15T21:14:01.720 INFO:teuthology.orchestra.run.smithi104.stdout:Priority: optional 2024-09-15T21:14:01.720 INFO:teuthology.orchestra.run.smithi104.stdout:Section: kernel 2024-09-15T21:14:01.720 INFO:teuthology.orchestra.run.smithi104.stdout:Installed-Size: 21 2024-09-15T21:14:01.720 INFO:teuthology.orchestra.run.smithi104.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T21:14:01.720 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: amd64 2024-09-15T21:14:01.720 INFO:teuthology.orchestra.run.smithi104.stdout:Source: linux-meta 2024-09-15T21:14:01.720 INFO:teuthology.orchestra.run.smithi104.stdout:Version: 5.15.0.121.121 2024-09-15T21:14:01.721 INFO:teuthology.orchestra.run.smithi104.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-15T21:14:01.721 INFO:teuthology.orchestra.run.smithi104.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T21:14:01.721 INFO:teuthology.orchestra.run.smithi104.stdout:Recommends: thermald 2024-09-15T21:14:01.721 INFO:teuthology.orchestra.run.smithi104.stdout:Description: Generic Linux kernel image 2024-09-15T21:14:01.721 INFO:teuthology.orchestra.run.smithi104.stdout: This package will always depend on the latest generic kernel image 2024-09-15T21:14:01.721 INFO:teuthology.orchestra.run.smithi104.stdout: available. 2024-09-15T21:14:01.721 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-15T21:14:01.721 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T21:14:01.721 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T21:14:01.722 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T21:14:01.722 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T21:14:01.722 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get clean 2024-09-15T21:14:01.897 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-09-15T21:14:02.028 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-15T21:14:02.148 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T21:14:02.229 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-15T21:14:02.310 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-15T21:14:03.483 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-15T21:14:03.502 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T21:14:03.565 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-15T21:14:03.772 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-09-15T21:14:03.774 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-09-15T21:14:03.962 INFO:teuthology.orchestra.run.smithi104.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-15T21:14:03.962 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T21:14:03.962 INFO:teuthology.orchestra.run.smithi104.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T21:14:03.962 INFO:teuthology.orchestra.run.smithi104.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T21:14:03.962 INFO:teuthology.orchestra.run.smithi104.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T21:14:03.963 INFO:teuthology.orchestra.run.smithi104.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-15T21:14:03.963 INFO:teuthology.orchestra.run.smithi104.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-15T21:14:03.963 INFO:teuthology.orchestra.run.smithi104.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T21:14:04.004 INFO:teuthology.orchestra.run.smithi104.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-15T21:14:04.006 DEBUG:teuthology.orchestra.run.smithi104:> dpkg -s linux-image-generic 2024-09-15T21:14:04.021 INFO:teuthology.orchestra.run.smithi104.stdout:Package: linux-image-generic 2024-09-15T21:14:04.021 INFO:teuthology.orchestra.run.smithi104.stdout:Status: install ok installed 2024-09-15T21:14:04.021 INFO:teuthology.orchestra.run.smithi104.stdout:Priority: optional 2024-09-15T21:14:04.021 INFO:teuthology.orchestra.run.smithi104.stdout:Section: kernel 2024-09-15T21:14:04.021 INFO:teuthology.orchestra.run.smithi104.stdout:Installed-Size: 21 2024-09-15T21:14:04.021 INFO:teuthology.orchestra.run.smithi104.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T21:14:04.022 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: amd64 2024-09-15T21:14:04.022 INFO:teuthology.orchestra.run.smithi104.stdout:Source: linux-meta 2024-09-15T21:14:04.022 INFO:teuthology.orchestra.run.smithi104.stdout:Version: 5.15.0.121.121 2024-09-15T21:14:04.022 INFO:teuthology.orchestra.run.smithi104.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-15T21:14:04.022 INFO:teuthology.orchestra.run.smithi104.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T21:14:04.022 INFO:teuthology.orchestra.run.smithi104.stdout:Recommends: thermald 2024-09-15T21:14:04.022 INFO:teuthology.orchestra.run.smithi104.stdout:Description: Generic Linux kernel image 2024-09-15T21:14:04.022 INFO:teuthology.orchestra.run.smithi104.stdout: This package will always depend on the latest generic kernel image 2024-09-15T21:14:04.022 INFO:teuthology.orchestra.run.smithi104.stdout: available. 2024-09-15T21:14:04.022 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-09-15T21:14:04.066 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.pgsB7AUNPu 2024-09-15T21:14:04.066 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub/grub.cfg /tmp/tmp.pgsB7AUNPu 2024-09-15T21:14:04.123 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.pgsB7AUNPu 2024-09-15T21:14:04.240 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.pgsB7AUNPu is 10KB 2024-09-15T21:14:04.253 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.pgsB7AUNPu 2024-09-15T21:14:04.259 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-15T21:14:04.315 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T21:14:04.316 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-15T21:14:04.316 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-15T21:14:04.381 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-15T21:14:04.382 DEBUG:teuthology.orchestra.run.smithi104:> sudo update-grub 2024-09-15T21:14:05.577 INFO:teuthology.orchestra.run.smithi104.stderr:Sourcing file `/etc/default/grub' 2024-09-15T21:14:05.578 INFO:teuthology.orchestra.run.smithi104.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T21:14:05.580 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-09-15T21:14:05.788 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-15T21:14:05.794 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-15T21:14:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-15T21:14:06.052 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-15T21:14:06.093 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-15T21:14:06.097 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-15T21:14:06.231 INFO:teuthology.orchestra.run.smithi104.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-15T21:14:06.231 INFO:teuthology.orchestra.run.smithi104.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-15T21:14:06.231 INFO:teuthology.orchestra.run.smithi104.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-15T21:14:06.247 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-09-15T21:14:06.257 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-09-15T21:14:36.289 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T21:14:36.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T21:14:36.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:15:10.781 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-09-15T21:15:19.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T21:15:19.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:15:38.173 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-09-15T21:15:50.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T21:15:50.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:15:50.902 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-09-15T21:15:51.611 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T21:15:51.611 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-15T21:15:51.611 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-09-15T21:15:51.617 INFO:teuthology.orchestra.run.smithi104.stdout:5.15.0-121-generic 2024-09-15T21:15:51.617 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-15T21:15:51.617 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T21:15:51.617 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-15T21:15:52.618 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T21:15:52.618 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T21:15:52.801 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-09-15T21:15:52.845 DEBUG:teuthology.parallel:result is None 2024-09-15T21:15:52.846 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T21:15:52.863 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T21:15:52.863 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T21:15:52.869 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T21:15:52.885 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T21:15:52.919 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T21:15:52.920 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T21:15:52.939 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T21:15:52.953 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T21:15:52.953 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T21:15:52.995 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:15:53.002 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:15:53.003 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T21:15:53.010 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T21:15:53.010 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T21:15:53.061 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T21:15:53.071 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T21:15:53.071 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T21:15:53.112 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T21:15:53.160 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T21:15:53.208 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T21:15:53.208 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T21:15:53.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-09-15T21:15:53.361 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T21:15:53.368 INFO:teuthology.task.internal:Starting timer... 2024-09-15T21:15:53.368 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T21:15:53.399 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T21:15:53.431 DEBUG:teuthology.task.selinux:Excluding smithi104: OS 'ubuntu' does not support SELinux 2024-09-15T21:15:53.431 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T21:15:53.431 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-15T21:15:53.431 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T21:15:53.431 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T21:15:53.464 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-15T21:15:53.467 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-15T21:15:53.612 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T21:15:53.623 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T21:15:53.624 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi104.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T21:23:21.055 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi104.front.sepia.ceph.com')] 2024-09-15T21:23:21.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T21:23:21.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:23:21.141 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-09-15T21:23:21.218 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-15T21:23:21.218 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T21:23:21.229 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T21:23:21.229 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T21:23:21.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T21:23:21.289 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-15T21:23:21.289 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: Command line: ntpd -gq 2024-09-15T21:23:21.289 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: ---------------------------------------------------- 2024-09-15T21:23:21.289 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: ntp-4 is maintained by Network Time Foundation, 2024-09-15T21:23:21.289 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-15T21:23:21.289 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: corporation. Support and training for ntp-4 are 2024-09-15T21:23:21.290 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: available at https://www.nwtime.org/support 2024-09-15T21:23:21.290 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: ---------------------------------------------------- 2024-09-15T21:23:21.290 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: proto: precision = 0.045 usec (-24) 2024-09-15T21:23:21.290 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: basedate set to 2022-02-04 2024-09-15T21:23:21.290 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: gps base set to 2022-02-06 (week 2196) 2024-09-15T21:23:21.290 INFO:teuthology.orchestra.run.smithi104.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T21:23:21.291 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T21:23:21.291 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: restrict ::: KOD does nothing without LIMITED. 2024-09-15T21:23:21.291 INFO:teuthology.orchestra.run.smithi104.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-15T21:23:21.292 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: Listen and drop on 0 v6wildcard [::]:123 2024-09-15T21:23:21.292 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-15T21:23:21.292 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: Listen normally on 2 lo 127.0.0.1:123 2024-09-15T21:23:21.292 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: Listen normally on 3 enp3s0f1 172.21.15.104:123 2024-09-15T21:23:21.292 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: Listen normally on 4 lo [::1]:123 2024-09-15T21:23:21.292 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce89%5]:123 2024-09-15T21:23:21.293 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:21 ntpd[18485]: Listening on routing socket on fd #22 for interface updates 2024-09-15T21:23:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:22 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:22 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T21:23:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T21:23:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T21:23:25.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:25 ntpd[18485]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T21:23:26.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:26 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:26.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:26 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:26.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:26 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:26.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:26 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:26.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:26 ntpd[18485]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T21:23:26.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:26 ntpd[18485]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T21:23:27.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:27.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:27.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:27.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:27.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:27.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:27.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T21:23:27.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T21:23:27.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:27.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:27.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T21:23:27.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:27 ntpd[18485]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T21:23:28.293 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:28 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:28.293 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:28 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:28.293 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:28 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:28.293 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:28 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:28.294 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:28 ntpd[18485]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T21:23:28.294 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:28 ntpd[18485]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T21:23:29.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:29.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:29.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:29.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:29.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:29.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:29.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T21:23:29.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T21:23:29.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:29.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:29.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T21:23:29.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:29 ntpd[18485]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T21:23:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:30 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:30 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:30 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:30 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:30 ntpd[18485]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T21:23:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:30 ntpd[18485]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T21:23:31.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:31 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:31.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:31 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:31.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:31 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:31.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:31 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:31.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:31 ntpd[18485]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T21:23:31.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:31 ntpd[18485]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T21:23:31.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:31 ntpd[18485]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T21:23:31.290 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:31 ntpd[18485]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T21:23:31.290 INFO:teuthology.orchestra.run.smithi104.stdout:15 Sep 21:23:31 ntpd[18485]: ntpd: time slew -0.000656 s 2024-09-15T21:23:31.291 INFO:teuthology.orchestra.run.smithi104.stdout:ntpd: time slew -0.000656s 2024-09-15T21:23:31.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:31 ntpd[18485]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-15T21:23:31.291 INFO:teuthology.orchestra.run.smithi104.stderr:15 Sep 21:23:31 ntpd[18485]: can't open /var/log/ntpstats/loopstats.20240915: Permission denied 2024-09-15T21:23:31.344 INFO:teuthology.orchestra.run.smithi104.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T21:23:31.344 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================== 2024-09-15T21:23:31.344 INFO:teuthology.orchestra.run.smithi104.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T21:23:31.344 INFO:teuthology.orchestra.run.smithi104.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T21:23:31.345 INFO:teuthology.orchestra.run.smithi104.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T21:23:31.345 INFO:teuthology.orchestra.run.smithi104.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T21:23:31.346 INFO:teuthology.run_tasks:Running task install... 2024-09-15T21:23:31.355 DEBUG:teuthology.task.install:project ceph 2024-09-15T21:23:31.355 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}} 2024-09-15T21:23:31.355 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'} 2024-09-15T21:23:31.355 INFO:teuthology.task.install:Using flavor: default 2024-09-15T21:23:31.361 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T21:23:31.361 INFO:teuthology.task.install:extra packages: [] 2024-09-15T21:23:31.362 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-key list | grep Ceph 2024-09-15T21:23:31.446 INFO:teuthology.orchestra.run.smithi104.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-15T21:23:31.473 INFO:teuthology.orchestra.run.smithi104.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-15T21:23:31.473 INFO:teuthology.orchestra.run.smithi104.stdout:uid [ unknown] Ceph.com (release key) 2024-09-15T21:23:31.473 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-15T21:23:31.474 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T21:23:31.620 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default/ 2024-09-15T21:23:31.620 INFO:teuthology.task.install.deb:Package version is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T21:23:31.729 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T21:23:31.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-15T21:23:31.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-09-15T21:23:31.915 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T21:23:31.924 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-15T21:23:31.941 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-15T21:23:31.968 INFO:teuthology.orchestra.run.smithi104.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy InRelease 2024-09-15T21:23:31.989 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-15T21:23:32.011 INFO:teuthology.orchestra.run.smithi104.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-15T21:23:32.031 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,427 B] 2024-09-15T21:23:32.037 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-15T21:23:32.058 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-15T21:23:32.269 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 29.2 kB in 0s (84.5 kB/s) 2024-09-15T21:23:33.246 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-15T21:23:33.263 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4930-gfc3e0bff-1jammy cephadm=19.3.0-4930-gfc3e0bff-1jammy ceph-mds=19.3.0-4930-gfc3e0bff-1jammy ceph-mgr=19.3.0-4930-gfc3e0bff-1jammy ceph-common=19.3.0-4930-gfc3e0bff-1jammy ceph-fuse=19.3.0-4930-gfc3e0bff-1jammy ceph-test=19.3.0-4930-gfc3e0bff-1jammy ceph-volume=19.3.0-4930-gfc3e0bff-1jammy radosgw=19.3.0-4930-gfc3e0bff-1jammy python3-rados=19.3.0-4930-gfc3e0bff-1jammy python3-rgw=19.3.0-4930-gfc3e0bff-1jammy python3-cephfs=19.3.0-4930-gfc3e0bff-1jammy python3-rbd=19.3.0-4930-gfc3e0bff-1jammy libcephfs2=19.3.0-4930-gfc3e0bff-1jammy libcephfs-dev=19.3.0-4930-gfc3e0bff-1jammy librados2=19.3.0-4930-gfc3e0bff-1jammy librbd1=19.3.0-4930-gfc3e0bff-1jammy rbd-fuse=19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T21:23:33.318 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-15T21:23:33.503 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-09-15T21:23:33.504 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-09-15T21:23:33.688 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T21:23:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T21:23:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T21:23:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T21:23:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-15T21:23:33.689 INFO:teuthology.orchestra.run.smithi104.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-15T21:23:33.689 INFO:teuthology.orchestra.run.smithi104.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T21:23:33.689 INFO:teuthology.orchestra.run.smithi104.stdout:The following additional packages will be installed: 2024-09-15T21:23:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-15T21:23:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-15T21:23:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-15T21:23:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-15T21:23:33.690 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-15T21:23:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-15T21:23:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-15T21:23:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-15T21:23:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-15T21:23:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-15T21:23:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-15T21:23:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-15T21:23:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-15T21:23:33.691 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-15T21:23:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-15T21:23:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-15T21:23:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-15T21:23:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-15T21:23:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-15T21:23:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-15T21:23:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-15T21:23:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-15T21:23:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-15T21:23:33.692 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet zip 2024-09-15T21:23:33.693 INFO:teuthology.orchestra.run.smithi104.stdout:Suggested packages: 2024-09-15T21:23:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-15T21:23:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-15T21:23:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-15T21:23:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-15T21:23:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-15T21:23:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-15T21:23:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-15T21:23:33.693 INFO:teuthology.orchestra.run.smithi104.stdout:Recommended packages: 2024-09-15T21:23:33.693 INFO:teuthology.orchestra.run.smithi104.stdout: btrfs-tools 2024-09-15T21:23:33.795 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2024-09-15T21:23:33.795 INFO:teuthology.orchestra.run.smithi104.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-15T21:23:33.795 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-15T21:23:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-15T21:23:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-15T21:23:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-15T21:23:33.796 INFO:teuthology.orchestra.run.smithi104.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-15T21:23:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-15T21:23:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-15T21:23:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-15T21:23:33.797 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-15T21:23:33.798 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-15T21:23:33.799 INFO:teuthology.orchestra.run.smithi104.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-15T21:23:33.941 INFO:teuthology.orchestra.run.smithi104.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-15T21:23:33.941 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 244 MB of archives. 2024-09-15T21:23:33.941 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-15T21:23:33.941 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,633 kB] 2024-09-15T21:23:33.984 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-15T21:23:34.124 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,334 kB] 2024-09-15T21:23:34.171 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4930-gfc3e0bff-1jammy [721 kB] 2024-09-15T21:23:34.182 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4930-gfc3e0bff-1jammy [356 kB] 2024-09-15T21:23:34.188 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4930-gfc3e0bff-1jammy [32.8 kB] 2024-09-15T21:23:34.188 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4930-gfc3e0bff-1jammy [180 kB] 2024-09-15T21:23:34.191 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4930-gfc3e0bff-1jammy [71.3 kB] 2024-09-15T21:23:34.192 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4930-gfc3e0bff-1jammy [338 kB] 2024-09-15T21:23:34.198 INFO:teuthology.orchestra.run.smithi104.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4930-gfc3e0bff-1jammy [7,966 kB] 2024-09-15T21:23:34.330 INFO:teuthology.orchestra.run.smithi104.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4930-gfc3e0bff-1jammy [112 kB] 2024-09-15T21:23:34.331 INFO:teuthology.orchestra.run.smithi104.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4930-gfc3e0bff-1jammy [473 kB] 2024-09-15T21:23:34.338 INFO:teuthology.orchestra.run.smithi104.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4930-gfc3e0bff-1jammy [28.2 MB] 2024-09-15T21:23:34.408 INFO:teuthology.orchestra.run.smithi104.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-15T21:23:34.420 INFO:teuthology.orchestra.run.smithi104.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-15T21:23:34.501 INFO:teuthology.orchestra.run.smithi104.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-15T21:23:34.730 INFO:teuthology.orchestra.run.smithi104.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-15T21:23:34.741 INFO:teuthology.orchestra.run.smithi104.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-15T21:23:34.771 INFO:teuthology.orchestra.run.smithi104.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-15T21:23:34.779 INFO:teuthology.orchestra.run.smithi104.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-15T21:23:34.782 INFO:teuthology.orchestra.run.smithi104.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-15T21:23:34.783 INFO:teuthology.orchestra.run.smithi104.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-15T21:23:34.784 INFO:teuthology.orchestra.run.smithi104.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-15T21:23:34.802 INFO:teuthology.orchestra.run.smithi104.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-15T21:23:34.877 INFO:teuthology.orchestra.run.smithi104.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-15T21:23:34.938 INFO:teuthology.orchestra.run.smithi104.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-15T21:23:34.938 INFO:teuthology.orchestra.run.smithi104.stdout:Get:27 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4930-gfc3e0bff-1jammy [5,206 kB] 2024-09-15T21:23:34.941 INFO:teuthology.orchestra.run.smithi104.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-15T21:23:34.942 INFO:teuthology.orchestra.run.smithi104.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-15T21:23:34.946 INFO:teuthology.orchestra.run.smithi104.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-15T21:23:34.951 INFO:teuthology.orchestra.run.smithi104.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-15T21:23:34.954 INFO:teuthology.orchestra.run.smithi104.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-15T21:23:34.956 INFO:teuthology.orchestra.run.smithi104.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-15T21:23:34.957 INFO:teuthology.orchestra.run.smithi104.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-15T21:23:34.975 INFO:teuthology.orchestra.run.smithi104.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-15T21:23:35.002 INFO:teuthology.orchestra.run.smithi104.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4930-gfc3e0bff-1jammy [251 kB] 2024-09-15T21:23:35.004 INFO:teuthology.orchestra.run.smithi104.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [124 kB] 2024-09-15T21:23:35.006 INFO:teuthology.orchestra.run.smithi104.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4930-gfc3e0bff-1jammy [1,561 kB] 2024-09-15T21:23:35.019 INFO:teuthology.orchestra.run.smithi104.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4930-gfc3e0bff-1jammy [6,309 kB] 2024-09-15T21:23:35.041 INFO:teuthology.orchestra.run.smithi104.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-15T21:23:35.042 INFO:teuthology.orchestra.run.smithi104.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-15T21:23:35.045 INFO:teuthology.orchestra.run.smithi104.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-15T21:23:35.045 INFO:teuthology.orchestra.run.smithi104.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-15T21:23:35.045 INFO:teuthology.orchestra.run.smithi104.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-15T21:23:35.045 INFO:teuthology.orchestra.run.smithi104.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-15T21:23:35.050 INFO:teuthology.orchestra.run.smithi104.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-15T21:23:35.056 INFO:teuthology.orchestra.run.smithi104.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-15T21:23:35.057 INFO:teuthology.orchestra.run.smithi104.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-15T21:23:35.113 INFO:teuthology.orchestra.run.smithi104.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4930-gfc3e0bff-1jammy [23.1 MB] 2024-09-15T21:23:35.137 INFO:teuthology.orchestra.run.smithi104.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-15T21:23:35.139 INFO:teuthology.orchestra.run.smithi104.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-15T21:23:35.140 INFO:teuthology.orchestra.run.smithi104.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-15T21:23:35.141 INFO:teuthology.orchestra.run.smithi104.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-15T21:23:35.143 INFO:teuthology.orchestra.run.smithi104.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-15T21:23:35.145 INFO:teuthology.orchestra.run.smithi104.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-15T21:23:35.147 INFO:teuthology.orchestra.run.smithi104.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-15T21:23:35.218 INFO:teuthology.orchestra.run.smithi104.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-15T21:23:35.219 INFO:teuthology.orchestra.run.smithi104.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-15T21:23:35.220 INFO:teuthology.orchestra.run.smithi104.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-15T21:23:35.299 INFO:teuthology.orchestra.run.smithi104.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-15T21:23:35.302 INFO:teuthology.orchestra.run.smithi104.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-15T21:23:35.306 INFO:teuthology.orchestra.run.smithi104.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-15T21:23:35.308 INFO:teuthology.orchestra.run.smithi104.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-15T21:23:35.311 INFO:teuthology.orchestra.run.smithi104.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-15T21:23:35.313 INFO:teuthology.orchestra.run.smithi104.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-15T21:23:35.318 INFO:teuthology.orchestra.run.smithi104.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-15T21:23:35.380 INFO:teuthology.orchestra.run.smithi104.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-15T21:23:35.382 INFO:teuthology.orchestra.run.smithi104.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-15T21:23:35.387 INFO:teuthology.orchestra.run.smithi104.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-15T21:23:35.570 INFO:teuthology.orchestra.run.smithi104.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-15T21:23:35.570 INFO:teuthology.orchestra.run.smithi104.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [14.1 kB] 2024-09-15T21:23:35.653 INFO:teuthology.orchestra.run.smithi104.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [880 kB] 2024-09-15T21:23:35.653 INFO:teuthology.orchestra.run.smithi104.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-15T21:23:35.656 INFO:teuthology.orchestra.run.smithi104.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-15T21:23:35.657 INFO:teuthology.orchestra.run.smithi104.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-15T21:23:35.659 INFO:teuthology.orchestra.run.smithi104.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-15T21:23:35.665 INFO:teuthology.orchestra.run.smithi104.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4930-gfc3e0bff-1jammy [2,452 kB] 2024-09-15T21:23:35.698 INFO:teuthology.orchestra.run.smithi104.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4930-gfc3e0bff-1jammy [813 kB] 2024-09-15T21:23:35.704 INFO:teuthology.orchestra.run.smithi104.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-15T21:23:35.709 INFO:teuthology.orchestra.run.smithi104.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-15T21:23:35.709 INFO:teuthology.orchestra.run.smithi104.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4930-gfc3e0bff-1jammy [160 kB] 2024-09-15T21:23:35.709 INFO:teuthology.orchestra.run.smithi104.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-15T21:23:35.710 INFO:teuthology.orchestra.run.smithi104.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4930-gfc3e0bff-1jammy [61.4 MB] 2024-09-15T21:23:35.764 INFO:teuthology.orchestra.run.smithi104.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-15T21:23:35.764 INFO:teuthology.orchestra.run.smithi104.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-15T21:23:35.817 INFO:teuthology.orchestra.run.smithi104.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-15T21:23:35.818 INFO:teuthology.orchestra.run.smithi104.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-15T21:23:35.820 INFO:teuthology.orchestra.run.smithi104.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-15T21:23:35.821 INFO:teuthology.orchestra.run.smithi104.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-15T21:23:35.822 INFO:teuthology.orchestra.run.smithi104.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-15T21:23:35.824 INFO:teuthology.orchestra.run.smithi104.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-15T21:23:35.833 INFO:teuthology.orchestra.run.smithi104.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-15T21:23:35.898 INFO:teuthology.orchestra.run.smithi104.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-15T21:23:35.902 INFO:teuthology.orchestra.run.smithi104.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-15T21:23:35.904 INFO:teuthology.orchestra.run.smithi104.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-15T21:23:35.979 INFO:teuthology.orchestra.run.smithi104.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-15T21:23:35.987 INFO:teuthology.orchestra.run.smithi104.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-15T21:23:35.988 INFO:teuthology.orchestra.run.smithi104.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-15T21:23:35.990 INFO:teuthology.orchestra.run.smithi104.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-15T21:23:36.002 INFO:teuthology.orchestra.run.smithi104.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-15T21:23:36.003 INFO:teuthology.orchestra.run.smithi104.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-15T21:23:36.011 INFO:teuthology.orchestra.run.smithi104.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-15T21:23:36.061 INFO:teuthology.orchestra.run.smithi104.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-15T21:23:36.063 INFO:teuthology.orchestra.run.smithi104.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-15T21:23:36.064 INFO:teuthology.orchestra.run.smithi104.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-15T21:23:36.142 INFO:teuthology.orchestra.run.smithi104.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-15T21:23:36.144 INFO:teuthology.orchestra.run.smithi104.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-15T21:23:36.162 INFO:teuthology.orchestra.run.smithi104.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-15T21:23:36.163 INFO:teuthology.orchestra.run.smithi104.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-15T21:23:36.164 INFO:teuthology.orchestra.run.smithi104.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-15T21:23:36.170 INFO:teuthology.orchestra.run.smithi104.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-15T21:23:36.171 INFO:teuthology.orchestra.run.smithi104.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-15T21:23:36.942 INFO:teuthology.orchestra.run.smithi104.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4930-gfc3e0bff-1jammy [8,625 kB] 2024-09-15T21:23:37.049 INFO:teuthology.orchestra.run.smithi104.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4930-gfc3e0bff-1jammy [14.2 kB] 2024-09-15T21:23:37.050 INFO:teuthology.orchestra.run.smithi104.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4930-gfc3e0bff-1jammy [48.5 MB] 2024-09-15T21:23:38.092 INFO:teuthology.orchestra.run.smithi104.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4930-gfc3e0bff-1jammy [130 kB] 2024-09-15T21:23:38.094 INFO:teuthology.orchestra.run.smithi104.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4930-gfc3e0bff-1jammy [39.0 kB] 2024-09-15T21:23:38.095 INFO:teuthology.orchestra.run.smithi104.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4930-gfc3e0bff-1jammy [15.8 MB] 2024-09-15T21:23:38.393 INFO:teuthology.orchestra.run.smithi104.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [91.2 kB] 2024-09-15T21:23:38.944 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 244 MB in 5s (53.1 MB/s) 2024-09-15T21:23:39.113 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-15T21:23:39.215 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-15T21:23:39.221 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-15T21:23:39.312 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-15T21:23:39.623 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-15T21:23:39.639 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-15T21:23:39.673 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-15T21:23:39.985 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-15T21:23:40.011 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-15T21:23:40.051 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-15T21:23:40.513 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-15T21:23:40.539 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T21:23:40.588 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T21:23:40.941 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-15T21:23:40.967 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T21:23:40.999 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T21:23:41.327 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-15T21:23:41.353 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T21:23:41.393 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T21:23:41.746 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-15T21:23:41.773 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-15T21:23:41.817 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-15T21:23:42.153 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librados2. 2024-09-15T21:23:42.180 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../007-librados2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:23:42.220 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:42.714 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libnbd0. 2024-09-15T21:23:42.741 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-15T21:23:42.781 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-15T21:23:43.067 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librbd1. 2024-09-15T21:23:43.090 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../009-librbd1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:23:43.125 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:43.545 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libcephfs2. 2024-09-15T21:23:43.571 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:23:43.632 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:43.897 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rados. 2024-09-15T21:23:43.912 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:23:43.955 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:44.283 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-15T21:23:44.309 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T21:23:44.349 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:45.027 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cephfs. 2024-09-15T21:23:45.053 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:45.371 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-15T21:23:45.397 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T21:23:45.438 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:45.783 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-15T21:23:45.808 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-15T21:23:45.849 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-15T21:23:46.152 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-prettytable. 2024-09-15T21:23:46.178 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-15T21:23:46.210 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-15T21:23:46.437 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rbd. 2024-09-15T21:23:46.464 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:23:46.504 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:46.857 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-15T21:23:46.883 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-15T21:23:46.923 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-15T21:23:47.210 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-15T21:23:47.225 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-15T21:23:47.268 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-15T21:23:47.570 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-15T21:23:47.585 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-15T21:23:47.620 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-15T21:23:47.965 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua5.1. 2024-09-15T21:23:47.991 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-15T21:23:48.032 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-15T21:23:48.343 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua-any. 2024-09-15T21:23:48.369 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-15T21:23:48.410 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-15T21:23:48.646 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package zip. 2024-09-15T21:23:48.672 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-15T21:23:48.712 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking zip (3.0-12build2) ... 2024-09-15T21:23:49.090 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package unzip. 2024-09-15T21:23:49.116 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-15T21:23:49.165 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-15T21:23:49.594 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package luarocks. 2024-09-15T21:23:49.621 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-15T21:23:49.660 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-15T21:23:50.210 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librgw2. 2024-09-15T21:23:50.237 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../026-librgw2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:23:50.293 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:50.912 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rgw. 2024-09-15T21:23:50.939 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:23:50.987 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:51.348 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-15T21:23:51.374 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-15T21:23:51.415 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-15T21:23:51.709 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libradosstriper1. 2024-09-15T21:23:51.725 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:23:51.759 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:52.188 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-common. 2024-09-15T21:23:52.214 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:23:52.254 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:53.759 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-base. 2024-09-15T21:23:53.785 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:23:53.835 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:23:54.405 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-15T21:23:54.432 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-15T21:23:54.471 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-15T21:23:54.782 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cheroot. 2024-09-15T21:23:54.809 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-15T21:23:54.849 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-15T21:23:55.210 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-15T21:23:55.237 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-15T21:23:55.268 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-15T21:23:55.555 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-15T21:23:55.581 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-15T21:23:55.621 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-15T21:23:55.924 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-15T21:23:55.951 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-15T21:23:55.990 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-15T21:23:56.318 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-tempora. 2024-09-15T21:23:56.346 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-15T21:23:56.385 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-15T21:23:56.646 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-portend. 2024-09-15T21:23:56.672 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-15T21:23:56.713 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-15T21:23:57.040 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-15T21:23:57.067 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-15T21:23:57.107 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-15T21:23:57.410 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-15T21:23:57.437 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-15T21:23:57.476 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-15T21:23:57.813 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-natsort. 2024-09-15T21:23:57.840 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-15T21:23:57.880 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-15T21:23:58.149 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-logutils. 2024-09-15T21:23:58.165 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-15T21:23:58.210 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-15T21:23:58.519 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-15T21:23:58.545 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-15T21:23:58.585 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-15T21:23:58.888 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-mako. 2024-09-15T21:23:58.914 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-15T21:23:58.954 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-15T21:23:59.283 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-15T21:23:59.310 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-15T21:23:59.349 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-15T21:23:59.610 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-15T21:23:59.633 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-15T21:23:59.669 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-15T21:24:00.005 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-webob. 2024-09-15T21:24:00.031 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-15T21:24:00.071 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-15T21:24:00.441 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-waitress. 2024-09-15T21:24:00.468 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-15T21:24:00.510 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-15T21:24:00.819 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-tempita. 2024-09-15T21:24:00.846 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-15T21:24:00.885 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-15T21:24:01.163 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-paste. 2024-09-15T21:24:01.190 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-15T21:24:01.229 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-15T21:24:01.549 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-15T21:24:01.576 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-15T21:24:01.616 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-15T21:24:01.896 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-15T21:24:01.909 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-15T21:24:01.943 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-15T21:24:02.213 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-webtest. 2024-09-15T21:24:02.242 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-15T21:24:02.280 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-15T21:24:02.608 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pecan. 2024-09-15T21:24:02.635 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-15T21:24:02.674 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-15T21:24:03.046 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-certifi. 2024-09-15T21:24:03.073 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-15T21:24:03.112 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-15T21:24:03.432 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-idna. 2024-09-15T21:24:03.459 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-15T21:24:03.498 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-15T21:24:03.834 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-urllib3. 2024-09-15T21:24:03.861 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-15T21:24:03.901 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-15T21:24:04.229 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-requests. 2024-09-15T21:24:04.256 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-15T21:24:04.295 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-15T21:24:04.582 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-15T21:24:04.609 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-15T21:24:04.648 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-15T21:24:04.951 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-15T21:24:04.967 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T21:24:05.010 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:05.640 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-15T21:24:05.666 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:05.706 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:05.958 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-15T21:24:05.974 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-15T21:24:06.009 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-15T21:24:06.311 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-openssl. 2024-09-15T21:24:06.338 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-15T21:24:06.378 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-15T21:24:06.731 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr. 2024-09-15T21:24:06.757 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:06.797 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:08.085 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mon. 2024-09-15T21:24:08.112 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:08.159 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:08.888 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-osd. 2024-09-15T21:24:08.914 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:08.954 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:10.032 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph. 2024-09-15T21:24:10.059 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../067-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:10.107 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:10.535 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-fuse. 2024-09-15T21:24:10.562 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:10.609 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:11.046 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mds. 2024-09-15T21:24:11.073 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:11.121 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:11.549 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package cephadm. 2024-09-15T21:24:11.575 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../070-cephadm_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:11.615 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:11.976 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-15T21:24:12.003 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-15T21:24:12.051 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-15T21:24:12.379 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-15T21:24:12.395 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-15T21:24:12.429 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-15T21:24:13.467 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-babel. 2024-09-15T21:24:13.494 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-15T21:24:13.533 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-15T21:24:13.844 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jinja2. 2024-09-15T21:24:13.860 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-15T21:24:13.903 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-15T21:24:14.206 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-15T21:24:14.233 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T21:24:14.272 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:14.576 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-15T21:24:14.603 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-15T21:24:14.642 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-15T21:24:14.945 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-routes. 2024-09-15T21:24:14.972 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-15T21:24:15.011 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-15T21:24:15.348 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-15T21:24:15.375 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T21:24:15.414 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:18.110 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-15T21:24:18.127 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-15T21:24:18.169 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-15T21:24:18.647 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-joblib. 2024-09-15T21:24:18.675 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-15T21:24:18.713 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-15T21:24:19.100 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-15T21:24:19.128 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-15T21:24:19.167 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-15T21:24:19.469 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-sklearn. 2024-09-15T21:24:19.497 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-15T21:24:19.621 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-15T21:24:20.339 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-15T21:24:20.367 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T21:24:20.397 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:21.659 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cachetools. 2024-09-15T21:24:21.687 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-15T21:24:21.725 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-15T21:24:22.011 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-15T21:24:22.039 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-15T21:24:22.078 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-15T21:24:22.415 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-15T21:24:22.443 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-15T21:24:22.481 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-15T21:24:22.744 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rsa. 2024-09-15T21:24:22.773 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-15T21:24:22.803 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-15T21:24:23.123 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-google-auth. 2024-09-15T21:24:23.151 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-15T21:24:23.189 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-15T21:24:23.509 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-15T21:24:23.537 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-15T21:24:23.602 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-15T21:24:23.920 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-websocket. 2024-09-15T21:24:23.948 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-15T21:24:23.986 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-15T21:24:24.281 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-15T21:24:24.309 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-15T21:24:24.347 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-15T21:24:25.019 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-15T21:24:25.048 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T21:24:25.086 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:25.397 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-15T21:24:25.425 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-15T21:24:25.463 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-15T21:24:25.783 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-15T21:24:25.811 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-15T21:24:25.849 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-15T21:24:26.201 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package jq. 2024-09-15T21:24:26.218 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-15T21:24:26.268 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-15T21:24:26.521 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package socat. 2024-09-15T21:24:26.549 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-15T21:24:26.587 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-15T21:24:26.974 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package xmlstarlet. 2024-09-15T21:24:27.002 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-15T21:24:27.040 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-15T21:24:27.412 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-test. 2024-09-15T21:24:27.427 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:27.469 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:29.773 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-volume. 2024-09-15T21:24:29.801 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T21:24:29.839 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:30.134 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-15T21:24:30.162 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:30.237 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:30.529 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-15T21:24:30.557 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-15T21:24:30.595 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-15T21:24:30.874 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-15T21:24:30.890 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-15T21:24:30.924 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-15T21:24:31.303 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package nvme-cli. 2024-09-15T21:24:31.332 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-15T21:24:31.369 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-15T21:24:31.808 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package pkg-config. 2024-09-15T21:24:31.836 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-15T21:24:31.866 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-15T21:24:32.119 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-15T21:24:32.146 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-15T21:24:32.185 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-15T21:24:32.613 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-15T21:24:32.641 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-15T21:24:32.679 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-15T21:24:32.940 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pastescript. 2024-09-15T21:24:32.956 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-15T21:24:32.990 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-15T21:24:33.285 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pluggy. 2024-09-15T21:24:33.313 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-15T21:24:33.343 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-15T21:24:33.646 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-psutil. 2024-09-15T21:24:33.674 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-15T21:24:33.712 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-15T21:24:34.058 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-py. 2024-09-15T21:24:34.085 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-15T21:24:34.123 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-15T21:24:34.426 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pygments. 2024-09-15T21:24:34.454 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-15T21:24:34.502 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-15T21:24:34.947 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-15T21:24:34.975 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-15T21:24:35.013 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-15T21:24:35.324 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-toml. 2024-09-15T21:24:35.352 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-15T21:24:35.390 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-15T21:24:35.660 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pytest. 2024-09-15T21:24:35.688 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-15T21:24:35.726 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-15T21:24:36.088 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-simplejson. 2024-09-15T21:24:36.118 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-15T21:24:36.154 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-15T21:24:36.432 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-15T21:24:36.460 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-15T21:24:36.499 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-15T21:24:37.503 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package radosgw. 2024-09-15T21:24:37.531 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../117-radosgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:37.569 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:38.438 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package rbd-fuse. 2024-09-15T21:24:38.455 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T21:24:38.496 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:38.793 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-15T21:24:39.051 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-15T21:24:39.152 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-15T21:24:39.393 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-15T21:24:39.710 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-15T21:24:40.091 INFO:teuthology.orchestra.run.smithi104.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-15T21:24:40.179 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:40.771 INFO:teuthology.orchestra.run.smithi104.stdout:Adding system user cephadm....done 2024-09-15T21:24:41.021 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-15T21:24:41.382 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-15T21:24:41.756 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-15T21:24:42.038 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-15T21:24:42.289 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-15T21:24:42.549 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-15T21:24:42.675 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-15T21:24:42.977 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-15T21:24:43.328 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-15T21:24:43.580 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-15T21:24:43.847 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-15T21:24:44.124 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-15T21:24:44.375 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-15T21:24:44.709 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-15T21:24:44.951 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:45.203 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-15T21:24:45.312 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-15T21:24:45.438 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-15T21:24:45.564 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-15T21:24:45.690 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-15T21:24:45.940 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-15T21:24:46.346 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-15T21:24:46.371 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-15T21:24:46.435 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-15T21:24:46.561 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-15T21:24:46.921 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-15T21:24:47.047 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-15T21:24:47.348 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-15T21:24:47.615 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-15T21:24:47.925 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up zip (3.0-12build2) ... 2024-09-15T21:24:48.051 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-15T21:24:48.719 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-15T21:24:48.962 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-15T21:24:49.088 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-15T21:24:49.214 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-15T21:24:49.523 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-15T21:24:49.899 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-15T21:24:50.159 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-15T21:24:50.519 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-15T21:24:50.786 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-15T21:24:51.129 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-15T21:24:51.380 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-15T21:24:51.523 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:24:51.841 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-15T21:24:52.150 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-15T21:24:53.322 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-15T21:24:53.574 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T21:24:53.691 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-15T21:24:53.976 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-15T21:24:54.102 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-15T21:24:54.228 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-15T21:24:54.521 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-15T21:24:54.772 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-15T21:24:55.023 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T21:24:55.149 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-15T21:24:55.433 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-15T21:24:55.685 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-15T21:24:55.944 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-15T21:24:56.186 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-15T21:24:56.437 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-15T21:24:56.697 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-15T21:24:56.823 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-15T21:24:57.132 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-15T21:24:57.242 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-15T21:24:57.534 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-15T21:24:57.835 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-15T21:24:58.111 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-15T21:24:58.237 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-15T21:24:58.363 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-15T21:24:58.452 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-15T21:24:58.681 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-15T21:24:58.782 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-15T21:24:59.142 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-15T21:24:59.376 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-15T21:24:59.502 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-15T21:24:59.753 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T21:24:59.879 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-15T21:25:00.197 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-15T21:25:00.489 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-15T21:25:00.749 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-15T21:25:00.858 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-15T21:25:01.117 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-15T21:25:01.479 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-15T21:25:01.755 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-15T21:25:02.132 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-15T21:25:02.409 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-15T21:25:02.753 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-15T21:25:02.878 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:03.004 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:03.130 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-15T21:25:03.491 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:03.592 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:03.693 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-15T21:25:03.986 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:04.112 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:04.239 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:04.451 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T21:25:04.451 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T21:25:04.939 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:05.065 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:05.182 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:05.292 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-15T21:25:06.428 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:06.554 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:06.663 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:06.773 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:06.974 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:07.835 INFO:teuthology.orchestra.run.smithi104.stdout:Adding group ceph....done 2024-09-15T21:25:08.212 INFO:teuthology.orchestra.run.smithi104.stdout:Adding system user ceph....done 2024-09-15T21:25:08.355 INFO:teuthology.orchestra.run.smithi104.stdout:Setting system user ceph properties....done 2024-09-15T21:25:08.372 INFO:teuthology.orchestra.run.smithi104.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-15T21:25:08.507 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-15T21:25:08.838 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-15T21:25:09.603 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:09.712 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:10.147 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T21:25:10.147 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T21:25:10.621 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:10.968 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-15T21:25:11.448 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:11.661 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T21:25:11.661 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T21:25:12.108 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:12.302 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T21:25:12.302 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T21:25:12.979 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:13.368 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T21:25:13.368 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T21:25:13.830 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:13.957 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:14.099 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:14.295 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T21:25:14.295 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T21:25:14.759 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:14.902 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:15.028 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:15.170 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T21:25:15.514 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-15T21:25:15.615 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-15T21:25:15.751 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-15T21:25:16.825 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-15T21:25:17.497 INFO:teuthology.orchestra.run.smithi104.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-15T21:25:17.502 DEBUG:teuthology.parallel:result is None 2024-09-15T21:25:17.503 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T21:25:17.655 DEBUG:teuthology.orchestra.run.smithi104:> dpkg-query -W -f '${Version}' ceph 2024-09-15T21:25:17.674 INFO:teuthology.orchestra.run.smithi104.stdout:19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T21:25:17.674 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T21:25:17.675 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930-gfc3e0bff-1jammy is installed. 2024-09-15T21:25:17.677 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T21:25:17.678 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T21:25:17.678 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T21:25:17.731 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T21:25:17.731 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T21:25:17.731 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T21:25:17.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T21:25:17.844 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T21:25:17.845 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T21:25:17.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T21:25:17.897 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T21:25:17.954 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T21:25:17.954 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T21:25:17.954 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T21:25:18.009 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T21:25:18.065 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T21:25:18.079 INFO:tasks.workunit:Pulling workunits from ref 7498493279982dfe87f00616198a5967475f6169 2024-09-15T21:25:18.080 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T21:25:18.080 INFO:tasks.workunit:timeout=3h 2024-09-15T21:25:18.080 INFO:tasks.workunit:cleanup=True 2024-09-15T21:25:18.080 DEBUG:teuthology.orchestra.run.smithi104:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T21:25:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:25:18.115 INFO:teuthology.orchestra.run.smithi104.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-15T21:25:18.115 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T21:25:18.162 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-15T21:25:18.162 DEBUG:teuthology.orchestra.run.smithi104:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-15T21:25:18.214 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7498493279982dfe87f00616198a5967475f6169 2024-09-15T21:25:18.264 INFO:tasks.workunit.client.0.smithi104.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T21:26:11.091 INFO:tasks.workunit.client.0.smithi104.stderr:Note: switching to '7498493279982dfe87f00616198a5967475f6169'. 2024-09-15T21:26:11.091 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-09-15T21:26:11.091 INFO:tasks.workunit.client.0.smithi104.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T21:26:11.091 INFO:tasks.workunit.client.0.smithi104.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T21:26:11.091 INFO:tasks.workunit.client.0.smithi104.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T21:26:11.091 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-09-15T21:26:11.091 INFO:tasks.workunit.client.0.smithi104.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T21:26:11.091 INFO:tasks.workunit.client.0.smithi104.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T21:26:11.092 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-09-15T21:26:11.092 INFO:tasks.workunit.client.0.smithi104.stderr: git switch -c 2024-09-15T21:26:11.092 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-09-15T21:26:11.092 INFO:tasks.workunit.client.0.smithi104.stderr:Or undo this operation with: 2024-09-15T21:26:11.092 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-09-15T21:26:11.092 INFO:tasks.workunit.client.0.smithi104.stderr: git switch - 2024-09-15T21:26:11.092 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-09-15T21:26:11.092 INFO:tasks.workunit.client.0.smithi104.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T21:26:11.092 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-09-15T21:26:11.092 INFO:tasks.workunit.client.0.smithi104.stderr:HEAD is now at 74984932799 Merge pull request #59797 from zdover23/wip-doc-2024-09-15-install-install-storage-cluster 2024-09-15T21:26:11.094 DEBUG:teuthology.orchestra.run.smithi104:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T21:26:11.143 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-15T21:26:11.143 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T21:26:11.190 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2024-09-15T21:26:11.190 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2024-09-15T21:26:11.190 DEBUG:teuthology.orchestra.run.smithi104:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2024-09-15T21:26:11.243 INFO:tasks.workunit.client.0.smithi104.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-09-15T21:26:11.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ TIMEOUT=300 2024-09-15T21:26:11.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-15T21:26:11.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ MAX_TIMEOUT=15 2024-09-15T21:26:11.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ PG_NUM=4 2024-09-15T21:26:11.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ TMPDIR=/tmp 2024-09-15T21:26:11.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-15T21:26:11.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-15T21:26:11.244 INFO:tasks.workunit.client.0.smithi104.stderr:+++ uname 2024-09-15T21:26:11.244 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' Linux = FreeBSD ']' 2024-09-15T21:26:11.244 INFO:tasks.workunit.client.0.smithi104.stderr:++ SED=sed 2024-09-15T21:26:11.244 INFO:tasks.workunit.client.0.smithi104.stderr:++ AWK=awk 2024-09-15T21:26:11.245 INFO:tasks.workunit.client.0.smithi104.stderr:+++ stty -a 2024-09-15T21:26:11.245 INFO:tasks.workunit.client.0.smithi104.stderr:+++ head -1 2024-09-15T21:26:11.245 INFO:tasks.workunit.client.0.smithi104.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-15T21:26:11.252 INFO:tasks.workunit.client.0.smithi104.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T21:26:11.252 INFO:tasks.workunit.client.0.smithi104.stderr:++ termwidth= 2024-09-15T21:26:11.252 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-15T21:26:11.252 INFO:tasks.workunit.client.0.smithi104.stderr:++ DIFFCOLOPTS='-y ' 2024-09-15T21:26:11.252 INFO:tasks.workunit.client.0.smithi104.stderr:++ KERNCORE=kernel.core_pattern 2024-09-15T21:26:11.252 INFO:tasks.workunit.client.0.smithi104.stderr:++ EXTRA_OPTS= 2024-09-15T21:26:11.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ test '' = TESTS 2024-09-15T21:26:11.256 INFO:tasks.workunit.client.0.smithi104.stderr:++ uname -m 2024-09-15T21:26:11.258 INFO:tasks.workunit.client.0.smithi104.stderr:+ arch=x86_64 2024-09-15T21:26:11.258 INFO:tasks.workunit.client.0.smithi104.stderr:+ case $arch in 2024-09-15T21:26:11.258 INFO:tasks.workunit.client.0.smithi104.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2024-09-15T21:26:11.258 INFO:tasks.workunit.client.0.smithi104.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2024-09-15T21:26:11.258 INFO:tasks.workunit.client.0.smithi104.stderr:+ plugins=(jerasure shec lrc isa) 2024-09-15T21:26:11.258 INFO:tasks.workunit.client.0.smithi104.stderr:+ main test-erasure-code-plugins 2024-09-15T21:26:11.258 INFO:tasks.workunit.client.0.smithi104.stderr:+ local dir=td/test-erasure-code-plugins 2024-09-15T21:26:11.258 INFO:tasks.workunit.client.0.smithi104.stderr:+ shift 2024-09-15T21:26:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:+ shopt -s -o xtrace 2024-09-15T21:26:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T21:26:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T21:26:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T21:26:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T21:26:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T21:26:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T21:26:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T21:26:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T21:26:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T21:26:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/test-erasure-code-plugins 2024-09-15T21:26:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2024-09-15T21:26:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2024-09-15T21:26:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2024-09-15T21:26:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2024-09-15T21:26:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2024-09-15T21:26:11.260 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2024-09-15T21:26:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none ' 2024-09-15T21:26:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2024-09-15T21:26:11.265 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2024-09-15T21:26:11.265 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T21:26:11.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2024-09-15T21:26:11.265 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_preload_no_warning 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_preload_no_warning_default 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_preload_warning' 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:26:11.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:26:11.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:26:11.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:26:11.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:26:11.267 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:26:11.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:26:11.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:26:11.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:26:11.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:26:11.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:26:11.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:26:11.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:26:11.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:26:11.270 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:26:11.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:26:11.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:26:11.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:26:11.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:26:11.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:11.272 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:26:11.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:26:11.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:26:11.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:26:11.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:26:11.274 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:26:11.274 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:11.274 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:26:11.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:26:11.277 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:26:11.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T21:26:11.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T21:26:11.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:26:11.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:26:11.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2024-09-15T21:26:11.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:26:11.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:26:11.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:26:11.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:26:11.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:26:11.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-15T21:26:11.870 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:26:11.870 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:26:11.870 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:26:11.870 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:26:11.871 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:11.871 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:26:11.871 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:26:11.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:26:12.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:26:12.066 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:26:12.066 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:26:12.067 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:26:12.067 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:26:12.068 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:26:12.068 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:26:12.068 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:26:12.068 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:26:12.068 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:26:12.068 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:12.068 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:26:12.068 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:26:12.068 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:26:12.069 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:26:12.184 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:26:12.185 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:26:12.185 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:26:12.185 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:26:12.186 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:26:12.186 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:26:12.186 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:26:12.186 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:26:12.186 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:26:12.186 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:12.186 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:26:12.186 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:26:12.186 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:26:12.187 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:26:12.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:26:12.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:26:12.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:26:12.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:26:12.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:26:12.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:26:12.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:26:13.196 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:26:13.196 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:26:13.196 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:26:13.196 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:26:13.196 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:13.196 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:26:13.196 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:26:13.197 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:26:13.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:26:13.350 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2024-09-15T21:26:13.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-15T21:26:13.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2024-09-15T21:26:13.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:26:13.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:26:13.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:26:13.356 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:26:13.356 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:26:13.356 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:26:13.356 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:26:13.356 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:13.356 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:26:13.356 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:26:13.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:26:13.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:26:13.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:26:13.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:26:13.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:26:13.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:26:13.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:26:13.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:26:13.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:26:13.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:26:13.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:26:13.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:26:13.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:26:13.359 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:26:13.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e671d890-24ca-43fc-8278-4fd42dcaee58 2024-09-15T21:26:13.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e671d890-24ca-43fc-8278-4fd42dcaee58' 2024-09-15T21:26:13.359 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 e671d890-24ca-43fc-8278-4fd42dcaee58 2024-09-15T21:26:13.359 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:26:13.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD1UOdmu9w0FhAAcZsT5A8fo/xLEFLP+hoEyw== 2024-09-15T21:26:13.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD1UOdmu9w0FhAAcZsT5A8fo/xLEFLP+hoEyw=="}' 2024-09-15T21:26:13.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e671d890-24ca-43fc-8278-4fd42dcaee58 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:26:13.696 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:26:13.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:26:13.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1UOdmu9w0FhAAcZsT5A8fo/xLEFLP+hoEyw== --osd-uuid e671d890-24ca-43fc-8278-4fd42dcaee58 2024-09-15T21:26:13.809 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:13.803+0000 7fefb08677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:13.811 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:13.807+0000 7fefb08677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:13.812 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:13.807+0000 7fefb08677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:13.813 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:13.807+0000 7fefb08677c0 -1 bdev(0x556923721c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:26:13.813 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:13.811+0000 7fefb08677c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:26:20.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:26:20.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:26:20.769 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:26:20.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:26:20.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:26:21.011 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:26:21.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:26:21.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:26:21.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:26:21.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:26:21.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:26:21.041 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:21.039+0000 7f994f9a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:21.044 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:21.039+0000 7f994f9a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:21.045 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:21.043+0000 7f994f9a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:21.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:26:21.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:26:21.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:26:21.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:26:21.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:26:21.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:21.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:26:21.230 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:26:21.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:21.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:21.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:22.368 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:22.363+0000 7f994f9a87c0 -1 Falling back to public interface 2024-09-15T21:26:22.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:22.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:22.454 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:26:22.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:26:22.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:22.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:22.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:23.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:23.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:23.884 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:26:23.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:26:23.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:23.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:24.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:25.113 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:25.111+0000 7f994f9a87c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:26:25.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:25.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:25.321 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:26:25.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:26:25.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:25.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:25.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:26.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:26.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:26.879 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:26:26.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:26:26.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:26.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:27.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:28.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:28.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:28.338 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:26:28.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:26:28.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:28.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:28.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:29.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:29.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:29.891 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:26:29.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:26:29.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:29.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:30.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:31.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:31.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:31.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:26:31.366 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:26:31.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:31.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:31.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:32.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:32.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:32.868 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:26:32.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:26:32.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:32.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:33.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:34.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:34.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:34.296 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:26:34.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:26:34.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:34.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:34.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:35.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:35.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:35.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:26:35.877 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:26:35.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:35.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:36.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:37.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:37.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:37.346 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:26:37.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:26:37.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:37.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:37.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:38.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:38.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:38.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:26:38.980 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:26:38.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:38.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:40.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:41.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:41.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:41.005 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:26:41.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:26:41.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:41.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:41.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:42.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:42.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:42.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:26:42.558 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:26:42.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:42.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:43.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:44.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:44.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:44.359 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:26:44.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:26:44.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:44.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:45.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:46.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:46.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:46.690 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:26:46.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:26:46.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:46.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:48.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:49.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:49.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:49.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:26:49.386 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:26:49.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:49.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:50.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:51.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:51.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:51.121 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:26:51.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:26:51.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:51.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:51.570 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3825056999,v1:127.0.0.1:6803/3825056999] [v2:127.0.0.1:6804/3825056999,v1:127.0.0.1:6805/3825056999] exists,up e671d890-24ca-43fc-8278-4fd42dcaee58 2024-09-15T21:26:51.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:26:51.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:26:51.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:26:51.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-15T21:26:51.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2024-09-15T21:26:51.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:26:51.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/1 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:26:51.574 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:26:51.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:26:51.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:26:51.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:26:51.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:26:51.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:26:51.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/1 2024-09-15T21:26:51.576 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:26:51.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1e613743-fbc3-4c7f-bcee-3d90b77f195e 2024-09-15T21:26:51.577 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 1e613743-fbc3-4c7f-bcee-3d90b77f195e 2024-09-15T21:26:51.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1e613743-fbc3-4c7f-bcee-3d90b77f195e' 2024-09-15T21:26:51.577 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:26:51.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAbUedmIPhXIxAADJTDG27jKcNg7dHqka95Eg== 2024-09-15T21:26:51.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAbUedmIPhXIxAADJTDG27jKcNg7dHqka95Eg=="}' 2024-09-15T21:26:51.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1e613743-fbc3-4c7f-bcee-3d90b77f195e -i td/test-erasure-code-plugins/1/new.json 2024-09-15T21:26:52.079 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:26:52.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/1/new.json 2024-09-15T21:26:52.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbUedmIPhXIxAADJTDG27jKcNg7dHqka95Eg== --osd-uuid 1e613743-fbc3-4c7f-bcee-3d90b77f195e 2024-09-15T21:26:52.129 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:52.127+0000 7fafc88e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:52.151 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:52.147+0000 7fafc88e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:52.152 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:52.147+0000 7fafc88e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:52.152 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:52.147+0000 7fafc88e07c0 -1 bdev(0x55d17bae4800 td/test-erasure-code-plugins/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:26:52.152 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:52.147+0000 7fafc88e07c0 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2024-09-15T21:26:58.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2024-09-15T21:26:58.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:26:58.759 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T21:26:58.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:26:58.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:26:59.335 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T21:26:59.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:26:59.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:26:59.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:26:59.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:26:59.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:26:59.365 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:59.363+0000 7fe8944fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:59.379 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:59.375+0000 7fe8944fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:59.381 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:26:59.375+0000 7fe8944fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:59.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:26:59.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:26:59.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:26:59.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:26:59.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:26:59.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:59.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:26:59.823 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:26:59.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:59.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:00.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:00.900 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:00.894+0000 7fe8944fb7c0 -1 Falling back to public interface 2024-09-15T21:27:01.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:01.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:01.246 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:27:01.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:27:01.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:01.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:01.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:02.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:02.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:02.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:27:02.679 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:27:02.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:02.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:03.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:03.874 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:03.870+0000 7fe8944fb7c0 -1 osd.1 0 log_to_monitors true 2024-09-15T21:27:04.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:04.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:04.102 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:27:04.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:27:04.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:04.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:04.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:05.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:05.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:05.629 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:27:05.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:27:05.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:05.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:06.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:07.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:07.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:07.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:27:07.150 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:27:07.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:07.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:07.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:08.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:08.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:08.578 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:27:08.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:27:08.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:08.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:09.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:10.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:10.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:10.160 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:27:10.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:27:10.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:10.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:10.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:11.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:11.741 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:27:11.742 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:11.742 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:27:11.742 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:11.742 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:12.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:13.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:13.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:13.359 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:27:13.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:27:13.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:13.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:13.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:14.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:14.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:14.912 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:27:14.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:27:14.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:14.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:15.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:16.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:16.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:16.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:27:16.386 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:27:16.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:16.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:17.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:18.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:18.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:18.054 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:27:18.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:27:18.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:18.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:18.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:19.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:19.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:19.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:27:19.947 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:27:19.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:19.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:21.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:22.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:22.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:22.081 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:27:22.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:27:22.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:22.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:22.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:23.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:23.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:23.973 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:27:23.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:27:23.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:23.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:24.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:25.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:25.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:25.869 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:27:25.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:27:25.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:25.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:26.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:27.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:27.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:27.747 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:27:27.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:27:27.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:27.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:28.399 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2967882372,v1:127.0.0.1:6811/2967882372] [v2:127.0.0.1:6812/2967882372,v1:127.0.0.1:6813/2967882372] exists,up 1e613743-fbc3-4c7f-bcee-3d90b77f195e 2024-09-15T21:27:28.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:27:28.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:27:28.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:27:28.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-15T21:27:28.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2024-09-15T21:27:28.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:27:28.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:27:28.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:27:28.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:27:28.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/2 2024-09-15T21:27:28.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:27:28.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:27:28.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:27:28.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:27:28.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2024-09-15T21:27:28.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2024-09-15T21:27:28.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:27:28.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:27:28.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:27:28.404 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:27:28.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:27:28.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:27:28.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:27:28.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:27:28.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:27:28.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/2 2024-09-15T21:27:28.406 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:27:28.406 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 f478f770-7794-4b52-9c45-a32b0f60edea 2024-09-15T21:27:28.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f478f770-7794-4b52-9c45-a32b0f60edea 2024-09-15T21:27:28.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f478f770-7794-4b52-9c45-a32b0f60edea' 2024-09-15T21:27:28.407 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:27:28.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBAUedmCcoXGRAA4JMRfcU3lLQFZb+qCHgzEQ== 2024-09-15T21:27:28.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBAUedmCcoXGRAA4JMRfcU3lLQFZb+qCHgzEQ=="}' 2024-09-15T21:27:28.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f478f770-7794-4b52-9c45-a32b0f60edea -i td/test-erasure-code-plugins/2/new.json 2024-09-15T21:27:29.061 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:27:29.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/2/new.json 2024-09-15T21:27:29.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAUedmCcoXGRAA4JMRfcU3lLQFZb+qCHgzEQ== --osd-uuid f478f770-7794-4b52-9c45-a32b0f60edea 2024-09-15T21:27:29.116 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:29.110+0000 7fdb2916b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:27:29.126 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:29.122+0000 7fdb2916b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:27:29.127 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:29.122+0000 7fdb2916b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:27:29.128 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:29.122+0000 7fdb2916b7c0 -1 bdev(0x55eba028bc00 td/test-erasure-code-plugins/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:27:29.128 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:29.122+0000 7fdb2916b7c0 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2024-09-15T21:27:35.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2024-09-15T21:27:35.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:27:35.938 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T21:27:35.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:27:35.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:27:36.507 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T21:27:36.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:27:36.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:27:36.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:27:36.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:27:36.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:27:36.536 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:36.530+0000 7f59be8b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:27:36.548 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:36.542+0000 7f59be8b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:27:36.549 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:36.542+0000 7f59be8b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:27:36.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:27:36.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:27:36.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:27:36.977 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:27:36.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:27:36.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:27:36.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:36.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:27:36.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:36.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:37.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:38.045 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:38.038+0000 7f59be8b07c0 -1 Falling back to public interface 2024-09-15T21:27:38.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:38.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:38.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:27:38.449 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:27:38.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:38.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:38.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:39.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:39.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:39.866 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:27:39.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:27:39.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:39.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:40.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:40.942 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:27:40.938+0000 7f59be8b07c0 -1 osd.2 0 log_to_monitors true 2024-09-15T21:27:41.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:41.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:41.286 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:27:41.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:27:41.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:41.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:41.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:42.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:42.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:42.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:27:42.811 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:27:42.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:42.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:43.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:44.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:44.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:44.235 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:27:44.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:27:44.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:44.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:44.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:45.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:45.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:45.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:27:45.714 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:27:45.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:45.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:46.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:47.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:47.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:47.254 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:27:47.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:27:47.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:47.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:47.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:48.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:48.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:48.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:27:48.937 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:27:48.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:48.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:49.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:50.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:50.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:50.420 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:27:50.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:27:50.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:50.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:50.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:51.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:51.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:51.879 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:27:51.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:27:51.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:51.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:52.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:53.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:53.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:53.313 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:27:53.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:27:53.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:53.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:53.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:54.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:54.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:54.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:27:54.755 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:27:54.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:54.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:55.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:56.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:56.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:56.499 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:27:56.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:27:56.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:56.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:57.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:58.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:58.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:58.556 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:27:58.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:27:58.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:58.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:27:59.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:00.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:00.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:00.326 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:28:00.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:28:00.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:00.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:28:01.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:02.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:02.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:02.239 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:28:02.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:28:02.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:02.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:28:03.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:04.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:04.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:04.417 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:28:04.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:28:04.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:04.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:28:05.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:06.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:06.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:06.276 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:28:06.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:28:06.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:06.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:28:06.701 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4067999109,v1:127.0.0.1:6819/4067999109] [v2:127.0.0.1:6820/4067999109,v1:127.0.0.1:6821/4067999109] exists,up f478f770-7794-4b52-9c45-a32b0f60edea 2024-09-15T21:28:06.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:28:06.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:28:06.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:28:06.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2024-09-15T21:28:06.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:28:07.161 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' does not exist 2024-09-15T21:28:07.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:28:07.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:28:07.705 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T21:28:07.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:28:08.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:28:11.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2024-09-15T21:28:11.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:28:11.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:28:11.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:28:11.020 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:28:11.021 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:28:11.021 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:28:11.021 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:28:11.021 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:28:11.022 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:28:11.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:28:11.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:28:11.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:28:11.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:28:11.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:28:11.140 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:28:11.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:28:11.578 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:28:11.578 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T21:28:11.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:28:11.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:11.579 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:28:11.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-15T21:28:11.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-15T21:28:11.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-15T21:28:11.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:11.764 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:28:11.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-15T21:28:11.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-15T21:28:11.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672976' 2024-09-15T21:28:11.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:11.913 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:28:12.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-15T21:28:12.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-15T21:28:12.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672976 2-64424509447' 2024-09-15T21:28:12.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:12.060 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-15T21:28:12.060 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:12.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:28:12.061 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-15T21:28:12.062 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:12.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-15T21:28:12.062 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 21474836504 2024-09-15T21:28:12.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-15T21:28:12.063 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:12.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-15T21:28:12.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:28:13.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:28:13.503 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:13.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-15T21:28:13.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:13.948 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-15T21:28:13.948 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:13.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:28:13.951 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:13.951 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-15T21:28:13.952 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 42949672976 2024-09-15T21:28:13.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-15T21:28:13.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-15T21:28:13.952 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:28:14.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672976 2024-09-15T21:28:14.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:14.390 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-15T21:28:14.390 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:14.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:28:14.392 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-15T21:28:14.392 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:14.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-15T21:28:14.394 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 64424509447 2024-09-15T21:28:14.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-15T21:28:14.394 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:28:14.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-15T21:28:14.820 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:28:14.820 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:14.821 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:15.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:28:15.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:28:15.477 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:28:15.477 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:28:15.477 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:28:15.477 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:28:15.477 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:28:15.478 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:28:15.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:28:15.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:28:15.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:15.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:16.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:28:16.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:28:16.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:28:16.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-15T21:28:16.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2024-09-15T21:28:16.980 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T21:28:16.980 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:28:16.980 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:28:16.980 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:16.980 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:16.980 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:16.981 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:28:16.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T21:28:16.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:28:17.071 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:28:17.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:28:17.093 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:16.737+0000 7fc5216e9640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-15T21:28:17.093 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:16.957+0000 7fc5216e9640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-15T21:28:17.093 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:16.957+0000 7fc5216e9640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-15T21:28:17.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-15T21:28:17.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2024-09-15T21:28:17.628 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T21:28:17.628 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:28:17.628 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:28:17.629 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:17.629 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:17.629 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:17.629 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:28:17.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T21:28:17.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:28:17.729 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:28:17.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:28:17.751 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:17.381+0000 7fc5216e9640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-15T21:28:17.751 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:17.605+0000 7fc5216e9640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-15T21:28:17.751 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:17.605+0000 7fc5216e9640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-15T21:28:17.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-15T21:28:17.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2024-09-15T21:28:18.338 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T21:28:18.338 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:28:18.338 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:28:18.338 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:18.338 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:18.338 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:18.338 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:28:18.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T21:28:18.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:28:18.425 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:28:18.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:28:18.444 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:18.069+0000 7fc5216e9640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-15T21:28:18.444 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:18.313+0000 7fc5216e9640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-15T21:28:18.444 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:18.313+0000 7fc5216e9640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-15T21:28:18.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-15T21:28:18.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2024-09-15T21:28:19.026 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T21:28:19.026 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:28:19.027 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:28:19.027 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:19.027 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:19.027 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:19.027 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:28:19.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T21:28:19.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:28:19.128 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:28:19.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:28:19.149 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:18.725+0000 7fc5216e9640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-15T21:28:19.149 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:19.001+0000 7fc5216e9640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-15T21:28:19.149 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:19.001+0000 7fc5216e9640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-15T21:28:19.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-15T21:28:19.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2024-09-15T21:28:19.704 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T21:28:19.704 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:28:19.704 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:28:19.705 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:19.705 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:19.705 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:19.705 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:28:19.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T21:28:19.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:28:19.802 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:28:19.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:28:19.823 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:19.457+0000 7fc5216e9640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-15T21:28:19.823 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:19.681+0000 7fc5216e9640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-15T21:28:19.823 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:19.681+0000 7fc5216e9640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-15T21:28:19.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-15T21:28:19.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2024-09-15T21:28:20.493 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T21:28:20.493 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:28:20.493 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:28:20.495 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:20.495 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:20.495 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:20.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:28:20.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T21:28:20.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:28:20.582 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:28:20.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:28:20.603 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:20.109+0000 7fc5216e9640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-15T21:28:20.603 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:20.473+0000 7fc5216e9640 0 mon.a@0(leader).osd e31 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-15T21:28:20.604 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:28:20.473+0000 7fc5216e9640 0 mon.a@0(leader).osd e31 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-15T21:28:20.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2024-09-15T21:28:20.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:20.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:28:20.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:28:20.605 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:28:20.605 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:28:20.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:28:20.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:28:20.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:28:20.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:28:20.715 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:28:20.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:28:20.716 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:28:20.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:28:20.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:28:20.718 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:28:20.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:28:20.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:28:20.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:28:20.719 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:28:20.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:28:20.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:28:20.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:28:20.733 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:28:20.733 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:20.733 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:20.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:28:20.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:28:20.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:28:20.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-15T21:28:20.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2024-09-15T21:28:20.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:28:20.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:28:20.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:28:20.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:28:20.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:28:20.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:28:20.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:28:20.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:28:20.739 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:28:20.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:28:20.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:28:20.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:28:20.740 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:28:20.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:28:20.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:28:20.742 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:28:20.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:28:20.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:20.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:20.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:28:20.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:28:20.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:28:20.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:28:20.744 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:28:20.745 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:20.745 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:20.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:28:20.746 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:28:20.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:28:20.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:28:20.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:28:20.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2024-09-15T21:28:20.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:20.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:28:20.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:28:20.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:28:20.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:28:20.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2024-09-15T21:28:21.243 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:28:21.243 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:28:21.243 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:28:21.243 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:28:21.243 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:21.243 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:21.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:28:21.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2024-09-15T21:28:21.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:28:21.465 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:28:21.465 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:28:21.465 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:28:21.465 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:28:21.466 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:28:21.467 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:28:21.467 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:28:21.467 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:28:21.467 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:21.467 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:21.468 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:21.468 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:28:21.468 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:28:21.468 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:28:21.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:28:21.579 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:28:21.579 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:28:21.579 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:28:21.580 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:28:21.580 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:28:21.580 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:28:21.580 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:28:21.580 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:21.580 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:21.580 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:21.581 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:28:21.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:28:21.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:28:21.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:28:21.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:21.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:28:21.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:28:21.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:28:21.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:28:21.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:28:22.584 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:28:22.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:28:22.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:28:22.585 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:28:22.585 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:22.585 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:22.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:28:22.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:28:22.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:28:22.620 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-15T21:28:22.620 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:28:22.620 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:28:22.621 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:22.621 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:22.621 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:22.621 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:28:22.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T21:28:22.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:28:22.718 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:28:22.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2024-09-15T21:28:22.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:22.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:28:22.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:28:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:28:22.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:28:22.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:28:22.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:28:22.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:28:22.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:28:22.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2024-09-15T21:28:22.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:28:22.738 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:28:22.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=36e24256-6400-49a1-a98e-758a9b55d0d7 2024-09-15T21:28:22.739 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 36e24256-6400-49a1-a98e-758a9b55d0d7 2024-09-15T21:28:22.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 36e24256-6400-49a1-a98e-758a9b55d0d7' 2024-09-15T21:28:22.739 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:28:22.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB2UedmMknPLBAACdzTZCZA/UYMeo5kNPTOeg== 2024-09-15T21:28:22.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB2UedmMknPLBAACdzTZCZA/UYMeo5kNPTOeg=="}' 2024-09-15T21:28:22.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 36e24256-6400-49a1-a98e-758a9b55d0d7 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:28:23.074 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:28:23.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:28:23.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQB2UedmMknPLBAACdzTZCZA/UYMeo5kNPTOeg== --osd-uuid 36e24256-6400-49a1-a98e-758a9b55d0d7 2024-09-15T21:28:23.117 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:28:23.109+0000 7fc08883a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:23.151 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:28:23.145+0000 7fc08883a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:23.153 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:28:23.145+0000 7fc08883a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:23.153 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:28:23.145+0000 7fc08883a7c0 -1 bdev(0x55cf168a1c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:28:23.153 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:28:23.145+0000 7fc08883a7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:28:29.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:28:29.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:28:29.349 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:28:29.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:28:29.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:28:29.928 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:28:29.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:28:29.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2024-09-15T21:28:29.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:28:29.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:28:29.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:28:29.959 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:28:29.953+0000 7f59ed4407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:29.991 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:28:29.985+0000 7f59ed4407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:29.993 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:28:29.985+0000 7f59ed4407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:30.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:28:30.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:28:30.366 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:28:30.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:28:30.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:28:30.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:28:30.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:30.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:28:30.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:30.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:30.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:31.513 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:28:31.505+0000 7f59ed4407c0 -1 Falling back to public interface 2024-09-15T21:28:31.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:31.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:31.788 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:28:31.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:28:31.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:31.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:32.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:33.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:33.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:33.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:28:33.266 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:28:33.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:33.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:33.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:34.574 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:28:34.569+0000 7f59ed4407c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:28:34.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:34.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:34.684 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:28:34.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:28:34.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:34.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:35.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:36.258 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:28:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:28:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:36.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:37.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:37.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:37.728 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:28:37.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:28:37.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:37.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:38.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:39.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:39.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:39.159 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:28:39.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:28:39.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:39.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:39.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:40.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:40.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:40.685 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:28:40.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:28:40.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:40.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:41.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:42.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:42.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:42.115 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:28:42.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:28:42.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:42.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:42.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:43.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:43.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:43.694 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:28:43.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:28:43.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:43.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:44.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:45.145 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:28:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:28:45.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:45.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:45.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:46.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:46.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:46.685 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:28:46.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:28:46.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:46.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:47.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:48.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:48.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:48.452 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:28:48.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:28:48.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:48.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:49.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:50.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:50.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:50.401 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:28:50.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:28:50.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:50.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:51.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:52.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:52.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:52.243 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:28:52.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:28:52.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:52.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:53.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:54.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:54.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:54.296 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:28:54.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:28:54.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:54.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:55.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:56.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:56.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:56.238 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:28:56.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:28:56.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:56.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:57.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:58.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:58.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:58.239 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:28:58.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:28:58.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:58.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:58.709 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/221762609,v1:127.0.0.1:6803/221762609] [v2:127.0.0.1:6804/221762609,v1:127.0.0.1:6805/221762609] exists,up 36e24256-6400-49a1-a98e-758a9b55d0d7 2024-09-15T21:28:58.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:28:58.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:28:58.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:28:58.710 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-15T21:28:58.710 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:28:58.710 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:28:58.710 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:58.711 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:58.711 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:58.711 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-osd.0.asok 2024-09-15T21:28:58.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T21:28:58.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-osd.0.asok log flush 2024-09-15T21:28:58.824 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:28:58.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:28:58.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-15T21:28:58.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-15T21:28:58.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:58.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:28:58.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:28:58.843 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:28:58.843 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:28:58.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:28:58.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:28:58.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:28:58.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:28:58.952 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:28:58.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:28:58.953 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:28:58.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:28:58.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:28:58.954 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:28:58.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:28:58.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:28:58.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:28:58.956 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:28:58.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:28:58.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:28:58.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:28:58.963 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:28:58.963 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:58.963 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:58.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:28:58.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:28:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:28:58.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:28:58.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:28:58.967 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:28:58.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:28:58.968 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:28:58.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:28:58.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:28:58.969 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:28:58.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:28:58.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:28:58.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:28:58.970 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:28:58.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:28:58.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:28:58.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:28:58.973 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:28:58.973 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:58.973 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:58.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:28:58.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:28:58.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:28:58.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:28:58.975 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:28:58.975 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:58.975 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:58.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:28:58.977 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:28:58.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:28:58.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:28:58.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:28:58.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2024-09-15T21:28:58.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:58.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:28:58.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:28:58.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:28:58.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:28:58.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2024-09-15T21:28:59.457 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:28:59.457 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:28:59.457 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:28:59.458 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:28:59.458 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:59.458 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:59.458 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:28:59.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2024-09-15T21:28:59.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:28:59.670 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:28:59.670 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:28:59.670 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:28:59.670 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:28:59.671 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:28:59.672 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:28:59.672 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:28:59.672 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:28:59.672 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:59.672 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:59.672 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:59.672 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:28:59.673 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:28:59.673 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:28:59.786 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:28:59.786 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:28:59.786 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:28:59.786 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:28:59.787 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:28:59.787 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:28:59.787 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:28:59.787 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:28:59.787 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:28:59.787 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:59.787 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:28:59.788 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:28:59.788 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:28:59.788 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:28:59.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:28:59.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:28:59.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:28:59.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:28:59.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:28:59.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:28:59.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:29:00.808 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:29:00.808 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:29:00.808 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:29:00.809 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:29:00.809 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:00.809 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:00.809 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:29:00.810 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:29:00.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:29:00.843 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-15T21:29:00.844 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:29:00.844 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:29:00.844 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:29:00.844 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:00.844 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:00.844 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:29:00.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T21:29:00.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:29:00.943 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:29:00.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2024-09-15T21:29:00.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:29:00.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:29:00.957 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:29:00.958 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:29:00.958 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:29:00.959 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:29:00.959 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:00.959 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:29:00.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2024-09-15T21:29:00.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:29:00.961 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:29:00.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=791451cd-6629-407e-9860-a264644f8209 2024-09-15T21:29:00.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 791451cd-6629-407e-9860-a264644f8209' 2024-09-15T21:29:00.962 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 791451cd-6629-407e-9860-a264644f8209 2024-09-15T21:29:00.962 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:29:00.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCcUedmFvMMOhAABLwo6FOu9TWPBe1xED0Htg== 2024-09-15T21:29:00.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCcUedmFvMMOhAABLwo6FOu9TWPBe1xED0Htg=="}' 2024-09-15T21:29:00.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 791451cd-6629-407e-9860-a264644f8209 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:29:01.277 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:29:01.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:29:01.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQCcUedmFvMMOhAABLwo6FOu9TWPBe1xED0Htg== --osd-uuid 791451cd-6629-407e-9860-a264644f8209 2024-09-15T21:29:01.317 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:01.309+0000 7efe5c8157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:01.332 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:01.325+0000 7efe5c8157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:01.334 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:01.329+0000 7efe5c8157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:01.334 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:01.329+0000 7efe5c8157c0 -1 bdev(0x55733ccc3c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:29:01.334 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:01.329+0000 7efe5c8157c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:29:07.751 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:29:07.751 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:29:07.752 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:29:07.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:29:07.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:29:08.307 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:29:08.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:29:08.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2024-09-15T21:29:08.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:29:08.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:29:08.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:29:08.337 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:08.333+0000 7f443d80b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:08.371 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:08.365+0000 7f443d80b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:08.372 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:08.365+0000 7f443d80b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:08.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:29:08.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:29:08.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:29:08.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:29:08.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:29:08.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:08.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:29:08.754 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:29:08.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:08.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:09.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:09.464 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:09.457+0000 7f443d80b7c0 -1 Falling back to public interface 2024-09-15T21:29:10.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:10.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:10.194 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:29:10.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:29:10.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:10.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:10.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:11.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:11.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:11.631 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:29:11.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:29:11.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:11.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:12.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:12.310 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:12.305+0000 7f443d80b7c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:29:13.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:13.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:13.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:29:13.044 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:29:13.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:13.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:13.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:14.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:14.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:14.501 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:29:14.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:29:14.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:14.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:14.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:15.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:15.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:15.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:29:15.954 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:29:15.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:15.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:16.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:17.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:17.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:17.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:29:17.527 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:29:17.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:17.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:18.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:19.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:19.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:19.014 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:29:19.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:29:19.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:19.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:19.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:20.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:20.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:20.488 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:29:20.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:29:20.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:20.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:20.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:21.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:21.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:21.937 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:29:21.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:29:21.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:21.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:22.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:23.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:23.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:23.366 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:29:23.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:29:23.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:23.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:23.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:24.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:24.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:24.791 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:29:24.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:29:24.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:24.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:25.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:26.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:26.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:26.559 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:29:26.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:29:26.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:26.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:27.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:28.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:28.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:28.548 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:29:28.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:29:28.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:28.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:29.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:30.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:30.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:30.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:29:30.504 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:29:30.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:30.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:31.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:32.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:32.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:32.395 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:29:32.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:29:32.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:32.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:33.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:34.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:34.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:34.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:29:34.241 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:29:34.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:34.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:35.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:36.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:36.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:36.343 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:29:36.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:29:36.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:36.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:36.797 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/547212966,v1:127.0.0.1:6803/547212966] [v2:127.0.0.1:6804/547212966,v1:127.0.0.1:6805/547212966] exists,up 791451cd-6629-407e-9860-a264644f8209 2024-09-15T21:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-15T21:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:36.800 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-osd.0.asok 2024-09-15T21:29:36.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T21:29:36.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-osd.0.asok log flush 2024-09-15T21:29:36.911 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:29:36.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:29:36.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-15T21:29:36.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-15T21:29:36.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:29:36.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:29:36.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:29:36.931 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:29:36.931 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:29:36.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:29:36.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:29:36.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:29:37.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:29:37.040 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:29:37.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:29:37.040 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:29:37.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:29:37.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:29:37.042 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:29:37.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:29:37.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:29:37.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:29:37.043 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:29:37.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:29:37.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:29:37.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:29:37.052 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:29:37.052 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:37.052 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:37.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:29:37.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:29:37.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:29:37.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-15T21:29:37.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-15T21:29:37.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:29:37.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:29:37.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:29:37.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:29:37.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:29:37.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:29:37.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:29:37.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:29:37.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:29:37.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:29:37.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:29:37.057 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:29:37.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:29:37.057 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:29:37.058 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:29:37.058 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:29:37.059 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:29:37.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:29:37.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:29:37.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:29:37.060 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:29:37.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:29:37.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:29:37.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:29:37.063 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:29:37.063 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:37.063 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:37.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:29:37.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:29:37.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:29:37.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:29:37.065 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:29:37.065 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:37.065 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:37.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:29:37.066 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:29:37.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:29:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:29:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:29:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2024-09-15T21:29:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:29:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:29:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:29:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:29:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:29:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2024-09-15T21:29:37.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:29:37.546 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:29:37.546 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:29:37.547 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:29:37.547 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:37.547 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:37.547 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:29:37.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2024-09-15T21:29:37.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:29:37.755 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:29:37.755 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:29:37.755 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:29:37.755 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:29:37.756 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:29:37.757 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:29:37.757 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:29:37.757 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:29:37.757 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:29:37.757 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:37.757 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:37.757 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:29:37.757 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:29:37.758 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:29:37.876 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:29:37.876 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:29:37.876 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:29:37.877 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:29:37.878 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:29:37.878 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:29:37.878 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:29:37.878 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:29:37.878 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:29:37.878 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:37.878 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:37.878 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:29:37.878 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:29:37.878 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:29:37.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:29:37.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:29:37.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:29:37.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:29:37.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:29:37.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:29:37.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:29:38.905 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:29:38.905 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:29:38.905 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:29:38.905 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:29:38.905 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:38.906 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:38.906 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:29:38.906 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:29:38.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:29:38.939 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-15T21:29:38.939 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:29:38.939 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:29:38.939 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:29:38.939 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:38.939 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:38.939 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:29:38.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T21:29:38.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:29:39.037 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:29:39.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2024-09-15T21:29:39.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:29:39.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:29:39.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:29:39.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:29:39.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:29:39.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:29:39.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:29:39.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:29:39.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:29:39.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:29:39.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:29:39.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:29:39.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:29:39.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:29:39.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:29:39.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:29:39.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:29:39.055 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:29:39.055 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:39.055 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:29:39.056 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:29:39.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:29:39.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:29:39.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:29:39.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:29:39.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:29:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:29:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:29:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:29:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:29:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:29:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:29:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2024-09-15T21:29:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:29:39.057 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:29:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=02b93743-aa5d-410b-8cc2-61af52befea5 2024-09-15T21:29:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 02b93743-aa5d-410b-8cc2-61af52befea5' 2024-09-15T21:29:39.058 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 02b93743-aa5d-410b-8cc2-61af52befea5 2024-09-15T21:29:39.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:29:39.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDDUedmoR9nBBAA6+Lf0Sv/IC9dtb9ymwsgIA== 2024-09-15T21:29:39.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDDUedmoR9nBBAA6+Lf0Sv/IC9dtb9ymwsgIA=="}' 2024-09-15T21:29:39.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 02b93743-aa5d-410b-8cc2-61af52befea5 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:29:39.362 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:29:39.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:29:39.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQDDUedmoR9nBBAA6+Lf0Sv/IC9dtb9ymwsgIA== --osd-uuid 02b93743-aa5d-410b-8cc2-61af52befea5 2024-09-15T21:29:39.410 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:39.404+0000 7fe19666b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:39.429 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:39.420+0000 7fe19666b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:39.431 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:39.424+0000 7fe19666b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:39.431 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:39.424+0000 7fe19666b7c0 -1 bdev(0x55830c3efc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:29:39.431 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:39.424+0000 7fe19666b7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:29:45.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:29:45.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:29:45.398 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:29:45.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:29:45.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:29:45.981 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:29:45.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:29:45.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:29:45.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:29:45.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:29:45.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2024-09-15T21:29:46.011 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:46.004+0000 7f60614117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:46.042 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:46.036+0000 7f60614117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:46.043 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:46.036+0000 7f60614117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:46.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:29:46.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:29:46.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:29:46.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:29:46.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:29:46.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:46.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:29:46.412 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:29:46.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:46.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:46.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:47.072 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:47.064+0000 7f60614117c0 -1 Falling back to public interface 2024-09-15T21:29:47.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:47.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:47.932 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:29:47.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:29:47.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:47.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:48.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:49.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:49.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:49.343 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:29:49.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:29:49.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:49.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:49.751 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:29:49.744+0000 7f60614117c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:29:49.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:50.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:50.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:50.778 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:29:50.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:29:50.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:50.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:51.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:52.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:52.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:52.272 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:29:52.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:29:52.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:52.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:52.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:53.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:53.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:53.855 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:29:53.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:29:53.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:53.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:54.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:55.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:55.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:55.293 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:29:55.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:29:55.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:55.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:55.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:56.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:56.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:56.875 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:29:56.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:29:56.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:56.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:57.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:58.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:58.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:58.296 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:58.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:59.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:59.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:59.889 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:29:59.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:29:59.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:59.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:00.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:01.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:01.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:01.356 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:30:01.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:30:01.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:01.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:01.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:02.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:02.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:02.852 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:30:02.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:30:02.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:02.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:03.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:04.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:04.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:04.700 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:30:04.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:30:04.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:04.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:05.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:06.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:06.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:06.586 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:30:06.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:30:06.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:06.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:07.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:08.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:08.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:08.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:30:08.494 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:30:08.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:08.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:09.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:10.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:10.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:10.599 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:30:10.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:30:10.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:10.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:11.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:12.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:12.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:12.474 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:30:12.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:30:12.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:12.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:13.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:14.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:14.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:14.646 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:30:14.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:30:14.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:14.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:15.087 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2574032937,v1:127.0.0.1:6803/2574032937] [v2:127.0.0.1:6804/2574032937,v1:127.0.0.1:6805/2574032937] exists,up 02b93743-aa5d-410b-8cc2-61af52befea5 2024-09-15T21:30:15.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:30:15.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:30:15.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:30:15.088 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-15T21:30:15.089 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:30:15.089 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:30:15.089 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:30:15.089 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:15.089 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:15.089 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-osd.0.asok 2024-09-15T21:30:15.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T21:30:15.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-osd.0.asok log flush 2024-09-15T21:30:15.195 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:30:15.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:30:15.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-15T21:30:15.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-15T21:30:15.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:15.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:30:15.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:30:15.216 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:30:15.216 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:30:15.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:30:15.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:30:15.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:30:15.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:30:15.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:30:15.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:30:15.527 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:30:15.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:30:15.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:30:15.529 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:30:15.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:30:15.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:30:15.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:30:15.531 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:30:15.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:30:15.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:30:15.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:30:15.540 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:30:15.540 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:15.540 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:15.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:30:15.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:30:15.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:30:15.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-15T21:30:15.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-15T21:30:15.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:15.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:30:15.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:15.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:30:15.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:30:15.543 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:30:15.543 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:30:15.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:30:15.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:30:15.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:30:15.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:30:15.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:30:15.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:30:15.546 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:30:15.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:30:15.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:30:15.548 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:30:15.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:30:15.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:30:15.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:30:15.549 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:30:15.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:30:15.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:30:15.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:30:15.553 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:30:15.553 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:15.553 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:15.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:30:15.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:30:15.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:30:15.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:30:15.556 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:30:15.556 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:15.556 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:15.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:30:15.558 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:30:15.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:30:15.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:30:15.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:30:15.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2024-09-15T21:30:15.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:15.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:30:15.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:30:15.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:30:15.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:30:15.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2024-09-15T21:30:16.024 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:30:16.024 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:16.025 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:16.025 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:16.025 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:16.025 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:16.026 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:30:16.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2024-09-15T21:30:16.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:30:16.241 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:30:16.242 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:30:16.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:30:16.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:30:16.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:30:16.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:30:16.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:30:16.352 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:30:16.353 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:30:16.353 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:30:16.353 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:30:16.353 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:16.354 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:16.354 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:30:16.354 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:30:16.354 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:30:16.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:30:16.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:16.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:30:16.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:30:16.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:30:16.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:30:16.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:30:17.399 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:30:17.399 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:17.399 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:17.399 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:17.400 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:17.400 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:17.400 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:30:17.401 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:30:17.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:30:17.433 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-15T21:30:17.433 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:30:17.433 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:30:17.433 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:30:17.433 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:17.433 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:17.433 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:30:17.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T21:30:17.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:30:17.525 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:30:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2024-09-15T21:30:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:30:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:30:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:30:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:30:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:30:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:30:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:30:17.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:30:17.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:30:17.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:30:17.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:30:17.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:30:17.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:30:17.542 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:30:17.542 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:17.542 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:17.542 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:17.542 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:30:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:30:17.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:30:17.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2024-09-15T21:30:17.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:30:17.544 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:30:17.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2c7911cf-19c8-4e9b-ad5f-3eca79543497 2024-09-15T21:30:17.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2c7911cf-19c8-4e9b-ad5f-3eca79543497' 2024-09-15T21:30:17.545 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 2c7911cf-19c8-4e9b-ad5f-3eca79543497 2024-09-15T21:30:17.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:30:17.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDpUedmxPVaIRAAFybkU3dp+ZdBlMGz7FN1oA== 2024-09-15T21:30:17.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDpUedmxPVaIRAAFybkU3dp+ZdBlMGz7FN1oA=="}' 2024-09-15T21:30:17.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2c7911cf-19c8-4e9b-ad5f-3eca79543497 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:30:17.839 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:30:17.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:30:17.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQDpUedmxPVaIRAAFybkU3dp+ZdBlMGz7FN1oA== --osd-uuid 2c7911cf-19c8-4e9b-ad5f-3eca79543497 2024-09-15T21:30:17.886 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:17.880+0000 7fba7e0607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:17.899 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:17.892+0000 7fba7e0607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:17.901 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:17.892+0000 7fba7e0607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:17.901 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:17.892+0000 7fba7e0607c0 -1 bdev(0x55832833e800 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:30:17.901 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:17.896+0000 7fba7e0607c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:30:24.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:30:24.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:30:24.201 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:30:24.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:30:24.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:30:24.779 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:30:24.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:30:24.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:30:24.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:30:24.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:30:24.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2024-09-15T21:30:24.810 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:24.804+0000 7fba624d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:24.836 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:24.828+0000 7fba624d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:24.837 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:24.832+0000 7fba624d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:25.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:30:25.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:30:25.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:30:25.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:30:25.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:30:25.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:25.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:30:25.216 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:30:25.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:25.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:26.060 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:26.052+0000 7fba624d07c0 -1 Falling back to public interface 2024-09-15T21:30:26.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:26.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:26.652 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:30:26.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:30:26.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:26.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:27.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:28.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:28.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:28.089 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:30:28.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:30:28.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:28.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:28.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:28.936 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:28.928+0000 7fba624d07c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:30:29.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:29.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:29.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:30:29.527 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:30:29.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:29.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:30.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:31.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:31.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:31.007 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:30:31.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:30:31.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:31.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:31.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:32.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:32.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:32.455 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:30:32.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:30:32.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:32.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:32.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:33.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:33.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:33.895 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:30:33.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:30:33.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:33.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:34.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:35.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:35.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:35.322 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:30:35.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:30:35.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:35.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:35.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:36.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:36.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:36.861 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:30:36.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:30:36.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:36.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:37.331 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:38.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:38.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:38.333 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:30:38.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:30:38.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:38.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:38.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:39.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:39.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:39.839 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:30:39.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:30:39.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:39.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:40.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:41.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:41.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:41.274 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:30:41.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:30:41.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:41.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:41.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:42.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:42.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:42.726 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:30:42.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:30:42.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:42.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:43.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:44.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:44.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:44.481 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:30:44.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:30:44.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:44.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:45.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:46.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:46.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:46.603 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:30:46.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:30:46.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:46.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:47.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:48.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:48.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:48.373 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:30:48.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:30:48.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:48.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:49.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:50.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:50.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:50.664 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:30:50.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:30:50.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:50.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:51.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:52.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:52.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:52.813 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:30:52.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:30:52.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:52.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:53.266 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/36926998,v1:127.0.0.1:6803/36926998] [v2:127.0.0.1:6804/36926998,v1:127.0.0.1:6805/36926998] exists,up 2c7911cf-19c8-4e9b-ad5f-3eca79543497 2024-09-15T21:30:53.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:30:53.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:30:53.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:30:53.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-15T21:30:53.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:30:53.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:30:53.269 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:30:53.269 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:53.269 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:53.269 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-osd.0.asok 2024-09-15T21:30:53.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T21:30:53.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-osd.0.asok log flush 2024-09-15T21:30:53.374 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:30:53.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:30:53.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-15T21:30:53.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-15T21:30:53.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:53.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:30:53.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:30:53.396 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:30:53.396 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:30:53.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:30:53.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:30:53.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:30:53.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:30:53.504 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:30:53.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:30:53.506 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:30:53.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:30:53.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:30:53.507 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:30:53.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:30:53.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:30:53.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:30:53.508 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:30:53.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:30:53.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:30:53.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:30:53.515 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:30:53.515 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:53.515 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:53.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:30:53.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:30:53.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:30:53.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2024-09-15T21:30:53.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:30:53.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:30:53.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:30:53.519 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:30:53.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:30:53.520 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:30:53.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:30:53.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:30:53.521 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:30:53.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:30:53.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:30:53.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:30:53.522 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:30:53.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:30:53.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:30:53.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:30:53.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:30:53.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:53.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:53.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:30:53.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:30:53.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:30:53.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:30:53.527 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:30:53.527 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:53.527 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:53.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:30:53.529 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:30:53.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:30:53.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:30:53.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:30:53.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2024-09-15T21:30:53.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:53.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:30:53.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:30:53.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:30:53.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:30:53.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-15T21:30:53.983 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:30:53.983 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:53.983 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:53.984 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:53.984 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:53.984 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:53.984 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:30:53.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:30:54.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:30:54.245 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:30:54.245 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:30:54.245 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:30:54.245 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:30:54.246 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:30:54.246 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:30:54.246 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:30:54.247 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:30:54.247 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:30:54.247 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:54.247 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:54.247 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:30:54.247 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:30:54.247 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:30:54.362 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:30:54.362 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:30:54.362 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:30:54.362 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:30:54.362 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:30:54.363 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:30:54.363 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:30:54.364 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:30:54.364 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:30:54.364 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:54.364 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:54.364 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:30:54.364 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:30:54.364 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:30:54.474 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2024-09-15T21:30:54.474 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:30:54.474 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:30:54.474 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:30:54.474 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:54.474 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:54.475 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:30:54.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-15T21:30:54.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:30:54.577 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:30:54.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:30:54.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:54.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:30:54.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:30:54.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:30:54.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:30:54.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:30:55.089 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:30:55.089 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:55.089 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:55.090 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:55.090 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:55.090 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:55.090 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:30:55.091 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:30:55.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:30:55.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:30:55.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:30:55.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:30:55.123 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:30:55.123 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:55.123 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:55.123 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:55.123 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:55.123 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:30:55.123 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:30:55.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:30:55.126 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:30:55.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3084caee-fd79-48e9-94b3-3e692c8177c2 2024-09-15T21:30:55.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3084caee-fd79-48e9-94b3-3e692c8177c2' 2024-09-15T21:30:55.126 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 3084caee-fd79-48e9-94b3-3e692c8177c2 2024-09-15T21:30:55.126 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:30:55.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAPUudm/CVDCBAAugzOnWEkv4bKZvkdMCDG7Q== 2024-09-15T21:30:55.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAPUudm/CVDCBAAugzOnWEkv4bKZvkdMCDG7Q=="}' 2024-09-15T21:30:55.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3084caee-fd79-48e9-94b3-3e692c8177c2 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:30:55.456 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:30:55.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:30:55.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPUudm/CVDCBAAugzOnWEkv4bKZvkdMCDG7Q== --osd-uuid 3084caee-fd79-48e9-94b3-3e692c8177c2 2024-09-15T21:30:55.502 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:55.500+0000 7fdb35abf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:55.524 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:55.520+0000 7fdb35abf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:55.526 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:55.524+0000 7fdb35abf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:55.526 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:55.524+0000 7fdb35abf7c0 -1 bdev(0x55c973cfbc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:30:55.526 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:30:55.524+0000 7fdb35abf7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:31:01.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:31:01.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:31:01.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:31:01.857 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:31:01.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:31:02.466 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:31:02.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:31:02.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:31:02.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:31:02.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:31:02.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:31:02.490 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:02.488+0000 7f99271537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:02.492 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:02.488+0000 7f99271537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:02.493 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:02.492+0000 7f99271537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:02.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:31:02.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:31:02.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:31:02.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:31:02.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:31:02.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:02.900 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:31:02.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:31:02.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:02.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:03.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:03.792 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:03.788+0000 7f99271537c0 -1 Falling back to public interface 2024-09-15T21:31:04.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:04.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:04.359 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:31:04.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:31:04.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:04.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:04.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:05.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:05.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:05.907 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:31:05.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:31:05.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:05.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:06.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:06.686 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:06.683+0000 7f99271537c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:31:07.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:07.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:07.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:31:07.222 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:31:07.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:07.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:07.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:08.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:08.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:08.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:31:08.776 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:31:08.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:08.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:09.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:10.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:10.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:10.196 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:31:10.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:31:10.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:10.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:10.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:11.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:11.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:11.649 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:31:11.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:31:11.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:11.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:12.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:13.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:13.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:13.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:31:13.129 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:31:13.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:13.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:13.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:14.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:14.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:14.628 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:31:14.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:31:14.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:14.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:15.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:16.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:16.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:16.066 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:31:16.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:31:16.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:16.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:16.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:17.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:17.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:17.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:31:17.605 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:31:17.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:17.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:18.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:19.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:19.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:19.038 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:31:19.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:31:19.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:19.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:19.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:20.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:20.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:20.792 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:31:20.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:31:20.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:20.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:21.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:22.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:22.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:22.750 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:31:22.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:31:22.751 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:22.751 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:23.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:24.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:24.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:24.632 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:31:24.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:31:24.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:24.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:25.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:26.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:26.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:26.801 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:31:26.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:31:26.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:26.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:27.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:28.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:28.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:28.592 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:31:28.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:31:28.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:28.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:29.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:30.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:30.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:30.786 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:31:30.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:31:30.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:30.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:31.255 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3639291652,v1:127.0.0.1:6803/3639291652] [v2:127.0.0.1:6804/3639291652,v1:127.0.0.1:6805/3639291652] exists,up 3084caee-fd79-48e9-94b3-3e692c8177c2 2024-09-15T21:31:31.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:31:31.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:31:31.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:31:31.256 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2024-09-15T21:31:31.256 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:31:31.256 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:31:31.257 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:31:31.257 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:31.257 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:31:31.257 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-osd.0.asok 2024-09-15T21:31:31.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-15T21:31:31.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-osd.0.asok log flush 2024-09-15T21:31:31.354 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:31:31.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:31:31.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2024-09-15T21:31:31.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2024-09-15T21:31:31.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:31:31.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:31:31.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:31:31.372 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:31:31.372 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:31:31.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:31:31.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:31:31.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:31:31.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:31:31.481 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:31:31.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:31:31.482 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:31:31.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:31:31.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:31:31.484 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:31:31.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:31:31.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:31:31.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:31:31.485 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:31:31.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:31:31.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:31:31.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:31:31.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:31:31.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:31.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:31:31.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:31:31.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:31:31.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:31:31.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2024-09-15T21:31:31.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-15T21:31:31.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2024-09-15T21:31:31.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2024-09-15T21:31:31.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T21:31:31.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T21:31:31.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:31:31.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:31:31.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:31:31.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:31:31.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:31:31.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:31:31.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:31:31.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:31:31.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:31:31.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:31:31.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:31:31.496 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:31:31.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:31:31.496 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:31:31.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:31:31.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:31:31.497 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:31:31.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:31:31.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:31:31.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:31:31.499 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:31:31.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:31:31.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:31:31.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:31:31.501 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:31:31.501 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:31.501 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:31:31.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:31:31.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:31:31.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:31:31.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:31:31.503 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:31:31.504 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:31.504 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:31:31.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:31:31.505 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:31:31.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:31:31.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:31:31.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:31:31.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2024-09-15T21:31:31.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:31:31.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:31:31.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:31:31.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:31:31.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:31:31.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2024-09-15T21:31:31.969 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:31:31.969 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:31:31.969 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:31:31.970 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:31:31.970 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:31.970 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:31:31.971 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:31:31.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2024-09-15T21:31:32.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:31:32.223 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:31:32.223 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:31:32.223 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:31:32.224 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:31:32.224 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:31:32.224 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:31:32.224 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:31:32.224 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:31:32.225 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:31:32.225 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:32.225 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:31:32.225 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:31:32.225 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:31:32.225 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:31:32.342 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:31:32.343 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:31:32.343 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:31:32.343 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:31:32.343 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:31:32.343 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:31:32.343 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:31:32.343 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:31:32.344 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:31:32.344 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:32.344 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:31:32.344 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:31:32.344 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:31:32.344 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:31:32.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:31:32.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:31:32.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:31:32.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:31:32.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:31:32.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:31:32.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:31:33.359 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:31:33.359 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:31:33.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:31:33.360 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:31:33.360 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:33.360 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:31:33.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:31:33.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:31:33.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:31:33.394 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T21:31:33.394 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:31:33.395 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:31:33.395 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:31:33.395 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:33.395 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:31:33.395 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:31:33.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:31:33.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:31:33.494 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:31:33.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2024-09-15T21:31:33.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:31:33.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:31:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:31:33.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:31:33.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:31:33.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:31:33.510 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:31:33.510 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:33.510 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:31:33.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:31:33.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:31:33.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:31:33.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:31:33.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:31:33.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:31:33.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:31:33.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:31:33.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:31:33.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:31:33.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:31:33.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:31:33.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2024-09-15T21:31:33.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:31:33.512 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:31:33.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=591cab86-0768-4095-abc8-5d43c0dac6a4 2024-09-15T21:31:33.514 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 591cab86-0768-4095-abc8-5d43c0dac6a4 2024-09-15T21:31:33.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 591cab86-0768-4095-abc8-5d43c0dac6a4' 2024-09-15T21:31:33.514 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:31:33.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1UudmcCCpHxAA4rFVG8HSbww0wfPJte/H/A== 2024-09-15T21:31:33.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1UudmcCCpHxAA4rFVG8HSbww0wfPJte/H/A=="}' 2024-09-15T21:31:33.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 591cab86-0768-4095-abc8-5d43c0dac6a4 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:31:33.808 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:31:33.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:31:33.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQA1UudmcCCpHxAA4rFVG8HSbww0wfPJte/H/A== --osd-uuid 591cab86-0768-4095-abc8-5d43c0dac6a4 2024-09-15T21:31:33.860 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:33.855+0000 7fb4b4bb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:33.877 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:33.871+0000 7fb4b4bb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:33.878 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:33.875+0000 7fb4b4bb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:33.879 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:33.875+0000 7fb4b4bb27c0 -1 bdev(0x55cf9d385c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:31:33.879 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:33.875+0000 7fb4b4bb27c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:31:40.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:31:40.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:31:40.360 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:31:40.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:31:40.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:31:40.919 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:31:40.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:31:40.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2024-09-15T21:31:40.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:31:40.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:31:40.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:31:40.949 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:40.947+0000 7f2bafb057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:40.951 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:40.947+0000 7f2bafb057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:40.952 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:40.947+0000 7f2bafb057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:41.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:31:41.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:31:41.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:31:41.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:31:41.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:31:41.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:41.346 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:31:41.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:31:41.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:41.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:41.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:41.948 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:41.943+0000 7f2bafb057c0 -1 Falling back to public interface 2024-09-15T21:31:42.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:42.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:42.795 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:31:42.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:31:42.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:42.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:43.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:44.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:44.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:44.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:31:44.270 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:31:44.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:44.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:44.655 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:31:44.651+0000 7f2bafb057c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:31:44.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:45.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:45.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:45.698 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:31:45.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:31:45.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:45.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:46.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:47.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:47.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:47.158 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:31:47.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:31:47.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:47.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:47.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:48.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:48.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:48.629 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:31:48.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:31:48.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:48.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:49.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:50.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:50.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:50.057 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:31:50.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:31:50.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:50.058 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:50.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:51.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:51.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:51.620 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:31:51.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:31:51.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:51.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:52.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:53.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:53.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:53.106 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:31:53.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:31:53.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:53.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:53.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:54.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:54.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:54.636 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:31:54.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:31:54.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:54.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:55.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:56.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:56.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:56.087 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:31:56.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:31:56.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:56.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:56.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:57.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:57.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:57.609 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:31:57.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:31:57.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:57.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:58.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:59.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:59.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:59.245 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:31:59.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:31:59.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:59.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:00.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:01.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:01.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:01.029 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:32:01.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:32:01.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:01.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:02.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:03.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:03.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:03.162 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:32:03.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:32:03.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:03.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:04.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:05.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:05.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:05.172 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:32:05.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:32:05.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:05.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:06.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:07.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:07.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:07.090 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:32:07.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:32:07.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:07.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:07.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:08.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:08.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:08.968 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:32:08.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:32:08.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:08.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:09.389 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/958757602,v1:127.0.0.1:6803/958757602] [v2:127.0.0.1:6804/958757602,v1:127.0.0.1:6805/958757602] exists,up 591cab86-0768-4095-abc8-5d43c0dac6a4 2024-09-15T21:32:09.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:32:09.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:32:09.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:32:09.390 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T21:32:09.390 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:32:09.390 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:32:09.391 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:09.391 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:09.391 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:09.391 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-osd.0.asok 2024-09-15T21:32:09.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:32:09.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-osd.0.asok log flush 2024-09-15T21:32:09.488 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:32:09.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:32:09.503 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:31:32.007+0000 7f0ca9fabe00 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T21:32:09.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-15T21:32:09.506 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:31:41.947+0000 7f2bafb057c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T21:32:09.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T21:32:09.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:09.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:32:09.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:32:09.507 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:32:09.507 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:32:09.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:32:09.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:32:09.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:32:09.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:32:09.615 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:32:09.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:32:09.616 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:32:09.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:32:09.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:32:09.618 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:32:09.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:32:09.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:32:09.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:32:09.619 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:32:09.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:32:09.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:32:09.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:32:09.626 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:32:09.626 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:09.626 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:09.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:32:09.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:32:09.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:32:09.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T21:32:09.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T21:32:09.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:09.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:32:09.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:09.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:32:09.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:32:09.629 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:32:09.629 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:32:09.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:32:09.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:32:09.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:32:09.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:32:09.630 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:32:09.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:32:09.631 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:32:09.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:32:09.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:32:09.632 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:32:09.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:32:09.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:32:09.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:32:09.633 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:32:09.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:32:09.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:32:09.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:32:09.636 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:32:09.636 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:09.636 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:09.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:32:09.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:32:09.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:32:09.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:32:09.638 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:32:09.638 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:09.638 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:09.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:32:09.640 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:32:09.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:32:09.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:32:09.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:32:09.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2024-09-15T21:32:09.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:09.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:32:09.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:32:09.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:32:09.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:32:09.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2024-09-15T21:32:10.141 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:32:10.141 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:32:10.141 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:32:10.142 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:32:10.142 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:10.142 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:10.143 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:32:10.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2024-09-15T21:32:10.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:32:10.351 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:32:10.351 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:32:10.351 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:32:10.351 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:32:10.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:32:10.352 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:32:10.352 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:32:10.352 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:32:10.353 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:10.353 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:10.353 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:10.353 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:32:10.353 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:32:10.354 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:32:10.468 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:32:10.468 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:32:10.469 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:32:10.469 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:32:10.470 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:32:10.470 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:32:10.470 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:32:10.470 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:32:10.470 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:10.471 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:10.471 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:10.471 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:32:10.471 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:32:10.471 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:32:10.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:32:10.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:10.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:32:10.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:32:10.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:32:10.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:32:10.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:32:11.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:32:11.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:32:11.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:32:11.492 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:32:11.493 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:11.493 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:11.493 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:32:11.494 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:32:11.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:32:11.523 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T21:32:11.523 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:32:11.523 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:32:11.523 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:11.523 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:11.523 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:11.524 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:32:11.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:32:11.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:32:11.619 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:32:11.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2024-09-15T21:32:11.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:11.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:32:11.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:32:11.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:32:11.637 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:32:11.638 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:32:11.638 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:11.638 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:11.638 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:32:11.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:32:11.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:32:11.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:32:11.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:32:11.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:32:11.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:32:11.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:32:11.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:32:11.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:32:11.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:32:11.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:32:11.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2024-09-15T21:32:11.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:32:11.639 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:32:11.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cd15b9a1-9dac-43a9-851e-0148e53d1ec9 2024-09-15T21:32:11.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 cd15b9a1-9dac-43a9-851e-0148e53d1ec9' 2024-09-15T21:32:11.640 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 cd15b9a1-9dac-43a9-851e-0148e53d1ec9 2024-09-15T21:32:11.640 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:32:11.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBbUudmq9a5JhAALLSlppytF7pjId21LNcSbA== 2024-09-15T21:32:11.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBbUudmq9a5JhAALLSlppytF7pjId21LNcSbA=="}' 2024-09-15T21:32:11.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cd15b9a1-9dac-43a9-851e-0148e53d1ec9 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:32:11.955 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:32:11.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:32:11.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQBbUudmq9a5JhAALLSlppytF7pjId21LNcSbA== --osd-uuid cd15b9a1-9dac-43a9-851e-0148e53d1ec9 2024-09-15T21:32:12.010 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:12.007+0000 7f72d73307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:12.033 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:12.027+0000 7f72d73307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:12.034 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:12.031+0000 7f72d73307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:12.035 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:12.031+0000 7f72d73307c0 -1 bdev(0x556af5c83c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:32:12.035 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:12.031+0000 7f72d73307c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:32:18.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:32:18.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:32:18.194 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:32:18.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:32:18.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:32:18.757 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:32:18.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:32:18.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2024-09-15T21:32:18.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:32:18.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:32:18.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:32:18.787 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:18.783+0000 7f236f9c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:18.802 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:18.799+0000 7f236f9c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:18.804 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:18.799+0000 7f236f9c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:19.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:32:19.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:32:19.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:32:19.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:32:19.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:32:19.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:19.232 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:32:19.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:32:19.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:19.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:19.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:20.036 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:20.031+0000 7f236f9c77c0 -1 Falling back to public interface 2024-09-15T21:32:20.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:20.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:20.672 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:32:20.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:32:20.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:20.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:21.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:22.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:22.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:22.097 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:32:22.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:32:22.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:22.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:22.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:22.957 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:22.951+0000 7f236f9c77c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:32:23.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:23.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:23.516 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:32:23.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:32:23.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:23.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:24.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:25.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:25.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:25.083 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:32:25.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:32:25.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:25.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:25.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:26.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:26.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:26.554 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:32:26.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:32:26.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:26.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:27.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:28.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:28.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:28.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:32:28.114 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:32:28.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:28.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:28.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:29.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:29.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:29.553 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:32:29.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:32:29.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:29.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:30.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:31.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:31.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:31.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:32:31.018 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:32:31.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:31.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:31.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:32.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:32.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:32.462 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:32:32.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:32:32.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:32.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:33.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:34.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:34.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:34.062 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:32:34.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:32:34.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:34.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:34.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:35.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:35.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:35.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:32:35.506 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:32:35.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:35.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:37.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:37.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:37.257 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:32:37.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:32:37.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:37.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:38.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:39.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:39.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:39.411 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:32:39.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:32:39.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:39.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:40.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:41.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:41.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:41.400 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:32:41.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:32:41.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:41.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:42.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:43.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:43.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:43.129 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:32:43.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:32:43.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:43.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:44.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:45.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:45.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:45.299 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:32:45.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:32:45.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:45.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:46.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:47.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:47.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:47.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:32:47.446 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:32:47.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:47.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:47.927 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1109246973,v1:127.0.0.1:6803/1109246973] [v2:127.0.0.1:6804/1109246973,v1:127.0.0.1:6805/1109246973] exists,up cd15b9a1-9dac-43a9-851e-0148e53d1ec9 2024-09-15T21:32:47.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:32:47.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:32:47.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:32:47.928 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T21:32:47.929 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:32:47.929 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:32:47.929 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:47.929 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:47.930 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:47.930 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-osd.0.asok 2024-09-15T21:32:47.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:32:47.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-osd.0.asok log flush 2024-09-15T21:32:48.032 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:32:48.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:32:48.048 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:32:10.175+0000 7f1078a61e00 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T21:32:48.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-15T21:32:48.050 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:32:20.035+0000 7f236f9c77c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T21:32:48.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T21:32:48.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:48.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:32:48.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:32:48.051 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:32:48.051 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:32:48.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:32:48.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:32:48.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:32:48.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:32:48.160 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:32:48.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:32:48.160 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:32:48.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:32:48.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:32:48.162 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:32:48.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:32:48.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:32:48.163 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:32:48.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:32:48.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:32:48.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:32:48.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:32:48.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:32:48.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:48.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:48.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:32:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:32:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:32:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T21:32:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T21:32:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:32:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:32:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:32:48.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:32:48.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:32:48.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:32:48.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:32:48.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:32:48.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:32:48.173 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:32:48.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:32:48.174 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:32:48.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:32:48.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:32:48.176 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:32:48.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:32:48.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:32:48.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:32:48.177 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:32:48.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:32:48.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:32:48.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:32:48.179 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:32:48.179 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:48.179 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:48.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:32:48.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:32:48.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:32:48.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:32:48.181 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:32:48.182 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:48.182 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:48.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:32:48.183 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:32:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:32:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:32:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:32:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2024-09-15T21:32:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:32:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:32:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:32:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:32:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2024-09-15T21:32:48.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:32:48.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:32:48.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:32:48.635 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:32:48.635 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:48.635 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:48.636 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:32:48.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2024-09-15T21:32:48.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:32:48.873 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:32:48.873 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:32:48.873 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:32:48.873 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:32:48.874 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:32:48.874 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:32:48.874 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:32:48.874 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:32:48.874 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:48.874 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:48.874 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:48.874 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:32:48.874 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:32:48.875 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:32:48.989 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:32:48.989 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:32:48.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:32:48.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:32:48.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:32:48.991 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:32:48.991 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:32:48.991 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:32:48.991 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:48.992 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:48.992 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:48.992 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:32:48.992 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:32:48.992 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:32:49.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:32:49.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:49.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:32:49.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:32:49.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:32:49.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:32:49.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:32:50.016 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:32:50.016 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:32:50.016 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:32:50.017 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:32:50.017 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:50.017 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:50.017 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:32:50.018 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:32:50.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:32:50.051 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T21:32:50.051 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:32:50.051 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:32:50.051 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:50.051 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:50.051 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:50.052 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:32:50.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:32:50.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:32:50.148 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:32:50.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:32:50.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:32:50.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:32:50.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:32:50.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:32:50.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:32:50.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:32:50.164 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:32:50.164 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:50.164 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:32:50.165 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:32:50.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:32:50.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:32:50.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:32:50.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:32:50.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:32:50.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:32:50.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:32:50.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:32:50.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:32:50.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:32:50.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:32:50.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2024-09-15T21:32:50.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:32:50.166 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:32:50.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b0d8e188-33d8-4593-aa3b-a2fdf81edb7e 2024-09-15T21:32:50.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b0d8e188-33d8-4593-aa3b-a2fdf81edb7e' 2024-09-15T21:32:50.167 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 b0d8e188-33d8-4593-aa3b-a2fdf81edb7e 2024-09-15T21:32:50.167 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:32:50.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCCUudmUS/zChAAvYqE0D3XiGZwNNCSNcsk4w== 2024-09-15T21:32:50.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCCUudmUS/zChAAvYqE0D3XiGZwNNCSNcsk4w=="}' 2024-09-15T21:32:50.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b0d8e188-33d8-4593-aa3b-a2fdf81edb7e -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:32:50.474 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:32:50.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:32:50.490 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQCCUudmUS/zChAAvYqE0D3XiGZwNNCSNcsk4w== --osd-uuid b0d8e188-33d8-4593-aa3b-a2fdf81edb7e 2024-09-15T21:32:50.522 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:50.519+0000 7f6a7057b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:50.533 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:50.531+0000 7f6a7057b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:50.534 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:50.531+0000 7f6a7057b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:50.534 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:50.531+0000 7f6a7057b7c0 -1 bdev(0x564af6fdbc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:32:50.534 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:50.531+0000 7f6a7057b7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:32:56.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:32:56.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:32:56.830 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:32:56.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:32:56.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:32:57.477 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:32:57.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:32:57.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2024-09-15T21:32:57.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:32:57.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:32:57.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:32:57.505 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:57.498+0000 7fb57d5317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:57.507 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:57.502+0000 7fb57d5317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:57.508 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:57.502+0000 7fb57d5317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:57.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:32:57.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:32:57.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:32:57.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:32:57.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:32:57.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:57.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:32:57.958 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:32:57.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:57.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:58.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:58.756 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:32:58.750+0000 7fb57d5317c0 -1 Falling back to public interface 2024-09-15T21:32:59.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:59.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:59.456 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:32:59.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:32:59.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:59.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:59.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:00.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:00.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:00.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:33:00.941 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:33:00.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:00.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:01.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:01.583 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:33:01.578+0000 7fb57d5317c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:33:02.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:02.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:02.378 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:33:02.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:33:02.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:02.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:02.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:03.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:03.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:03.854 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:33:03.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:33:03.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:03.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:04.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:05.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:05.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:05.316 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:33:05.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:33:05.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:05.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:05.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:06.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:06.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:06.829 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:33:06.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:33:06.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:06.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:07.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:08.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:08.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:08.312 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:33:08.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:33:08.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:08.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:08.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:09.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:09.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:09.823 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:33:09.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:33:09.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:09.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:10.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:11.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:11.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:11.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:33:11.315 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:33:11.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:11.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:11.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:12.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:12.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:12.896 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:33:12.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:33:12.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:12.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:13.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:14.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:14.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:14.338 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:33:14.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:33:14.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:14.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:14.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:15.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:15.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:15.906 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:33:15.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:33:15.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:15.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:16.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:17.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:17.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:17.927 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:33:17.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:33:17.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:17.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:18.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:19.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:19.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:19.883 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:33:19.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:33:19.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:19.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:20.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:21.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:21.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:21.710 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:33:21.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:33:21.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:21.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:22.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:23.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:23.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:23.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:33:23.910 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:33:23.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:23.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:24.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:25.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:25.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:25.780 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:33:25.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:33:25.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:25.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:26.199 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2195741207,v1:127.0.0.1:6803/2195741207] [v2:127.0.0.1:6804/2195741207,v1:127.0.0.1:6805/2195741207] exists,up b0d8e188-33d8-4593-aa3b-a2fdf81edb7e 2024-09-15T21:33:26.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:33:26.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:33:26.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:33:26.201 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T21:33:26.201 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:33:26.201 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:33:26.201 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:26.201 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:26.201 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:33:26.201 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-osd.0.asok 2024-09-15T21:33:26.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:33:26.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-osd.0.asok log flush 2024-09-15T21:33:26.298 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:33:26.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:33:26.312 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:32:48.671+0000 7f916708de00 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T21:33:26.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-15T21:33:26.313 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:32:58.754+0000 7fb57d5317c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T21:33:26.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T21:33:26.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:33:26.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:33:26.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:33:26.316 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:33:26.316 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:33:26.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:33:26.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:33:26.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:33:26.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:33:26.424 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:33:26.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:33:26.425 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:33:26.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:33:26.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:33:26.426 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:33:26.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:33:26.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:33:26.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:33:26.427 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:33:26.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:33:26.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:33:26.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:33:26.434 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:33:26.434 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:26.434 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:33:26.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:33:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:33:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:33:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T21:33:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T21:33:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:33:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:33:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:33:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:33:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:33:26.436 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:33:26.437 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:33:26.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:33:26.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:33:26.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:33:26.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:33:26.437 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:33:26.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:33:26.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:33:26.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:33:26.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:33:26.439 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:33:26.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:33:26.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:33:26.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:33:26.441 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:33:26.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:33:26.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:33:26.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:33:26.444 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:33:26.444 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:26.444 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:33:26.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:33:26.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:33:26.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:33:26.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:33:26.445 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:33:26.445 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:26.445 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:33:26.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:33:26.447 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:33:26.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:33:26.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:33:26.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:33:26.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2024-09-15T21:33:26.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:33:26.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:33:26.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:33:26.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:33:26.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:33:26.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2024-09-15T21:33:26.941 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:33:26.941 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:33:26.941 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:33:26.942 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:33:26.942 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:26.942 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:33:26.942 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:33:26.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2024-09-15T21:33:27.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:33:27.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:33:27.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:33:27.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:33:27.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:33:27.136 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:33:27.136 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:33:27.136 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:33:27.136 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:33:27.136 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:27.136 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:27.136 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:33:27.136 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:33:27.137 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:33:27.137 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:33:27.255 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:33:27.255 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:33:27.255 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:33:27.255 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:33:27.256 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:33:27.257 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:33:27.257 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:33:27.257 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:33:27.257 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:27.257 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:27.257 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:33:27.257 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:33:27.257 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:33:27.257 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:33:27.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:33:27.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:33:27.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:33:27.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:33:27.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:33:27.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:33:27.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:33:28.263 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:33:28.263 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:33:28.263 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:33:28.264 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:33:28.264 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:28.264 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:33:28.264 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:33:28.265 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:33:28.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:33:28.293 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T21:33:28.293 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:33:28.294 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:33:28.294 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:28.294 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:28.294 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:33:28.294 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:33:28.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:33:28.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:33:28.405 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:33:28.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:33:28.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:33:28.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:33:28.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:33:28.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:33:28.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:33:28.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:33:28.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:33:28.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:33:28.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:33:28.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2024-09-15T21:33:28.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:33:28.424 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:33:28.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7e2e4cef-db42-4dee-8551-2d5bca36ace8 2024-09-15T21:33:28.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7e2e4cef-db42-4dee-8551-2d5bca36ace8' 2024-09-15T21:33:28.425 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 7e2e4cef-db42-4dee-8551-2d5bca36ace8 2024-09-15T21:33:28.425 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:33:28.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCoUudmwhogGhAAzF+p7loxJKDOTImEmtnjDQ== 2024-09-15T21:33:28.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCoUudmwhogGhAAzF+p7loxJKDOTImEmtnjDQ=="}' 2024-09-15T21:33:28.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7e2e4cef-db42-4dee-8551-2d5bca36ace8 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:33:28.764 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:33:28.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:33:28.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQCoUudmwhogGhAAzF+p7loxJKDOTImEmtnjDQ== --osd-uuid 7e2e4cef-db42-4dee-8551-2d5bca36ace8 2024-09-15T21:33:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:33:28.806+0000 7fed96ac07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:28.850 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:33:28.846+0000 7fed96ac07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:28.851 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:33:28.846+0000 7fed96ac07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:28.852 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:33:28.846+0000 7fed96ac07c0 -1 bdev(0x5614ce567c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:33:28.852 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:33:28.846+0000 7fed96ac07c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:33:35.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:33:35.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:33:35.162 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:33:35.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:33:35.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:33:35.720 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:33:35.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:33:35.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2024-09-15T21:33:35.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:33:35.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:33:35.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:33:35.750 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:33:35.746+0000 7fbafb6fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:35.770 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:33:35.766+0000 7fbafb6fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:35.772 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:33:35.766+0000 7fbafb6fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:33:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:33:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:33:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:33:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:33:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:33:36.184 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:33:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:36.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:37.300 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:33:37.294+0000 7fbafb6fb7c0 -1 Falling back to public interface 2024-09-15T21:33:37.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:37.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:37.662 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:33:37.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:33:37.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:37.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:38.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:39.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:39.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:39.075 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:33:39.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:33:39.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:39.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:39.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:40.142 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:33:40.138+0000 7fbafb6fb7c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:33:40.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:40.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:40.498 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:33:40.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:33:40.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:40.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:40.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:41.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:41.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:41.926 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:33:41.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:33:41.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:41.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:42.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:43.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:43.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:43.361 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:33:43.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:33:43.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:43.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:43.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:44.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:44.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:44.926 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:33:44.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:33:44.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:44.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:45.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:46.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:46.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:46.363 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:33:46.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:33:46.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:46.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:46.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:47.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:47.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:47.942 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:33:47.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:33:47.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:47.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:48.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:49.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:49.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:49.383 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:33:49.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:33:49.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:49.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:49.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:50.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:50.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:50.994 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:33:50.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:33:50.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:50.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:51.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:52.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:52.466 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:33:52.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:52.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:33:52.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:52.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:52.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:53.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:53.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:53.946 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:33:53.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:33:53.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:53.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:54.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:55.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:55.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:55.813 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:33:55.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:33:55.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:55.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:56.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:57.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:57.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:57.882 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:33:57.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:33:57.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:57.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:58.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:59.742 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:59.742 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:59.742 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:33:59.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:33:59.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:59.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:00.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:01.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:01.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:01.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:34:01.306 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:34:01.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:01.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:02.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:03.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:03.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:03.134 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:34:03.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:34:03.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:03.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:03.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:04.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:04.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:04.632 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:34:04.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:34:04.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:04.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:05.066 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2710292295,v1:127.0.0.1:6803/2710292295] [v2:127.0.0.1:6804/2710292295,v1:127.0.0.1:6805/2710292295] exists,up 7e2e4cef-db42-4dee-8551-2d5bca36ace8 2024-09-15T21:34:05.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:34:05.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:34:05.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:34:05.070 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T21:34:05.070 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:34:05.070 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:34:05.070 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:34:05.071 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:05.071 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:05.072 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-osd.0.asok 2024-09-15T21:34:05.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:34:05.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-osd.0.asok log flush 2024-09-15T21:34:05.172 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:34:05.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:34:05.187 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:33:26.974+0000 7fa4ab61ae00 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T21:34:05.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-15T21:34:05.189 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:33:37.298+0000 7fbafb6fb7c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T21:34:05.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T21:34:05.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:05.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:34:05.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:34:05.191 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:34:05.191 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:34:05.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:34:05.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:34:05.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:34:05.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:34:05.299 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:34:05.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:34:05.300 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:34:05.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:34:05.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:34:05.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:34:05.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:34:05.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:34:05.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:34:05.303 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:34:05.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:34:05.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:34:05.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:34:05.310 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:34:05.310 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:05.310 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:05.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:34:05.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:34:05.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:34:05.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T21:34:05.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T21:34:05.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:05.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:34:05.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:05.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:34:05.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:34:05.313 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:34:05.313 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:34:05.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:34:05.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:34:05.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:34:05.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:34:05.314 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:34:05.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:34:05.314 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:34:05.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:34:05.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:34:05.315 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:34:05.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:34:05.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:34:05.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:34:05.317 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:34:05.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:34:05.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:34:05.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:34:05.319 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:34:05.319 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:05.319 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:05.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:34:05.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:34:05.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:34:05.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:34:05.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:34:05.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:05.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:05.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:34:05.323 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:34:05.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:34:05.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:34:05.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:34:05.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2024-09-15T21:34:05.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:05.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:34:05.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:34:05.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:34:05.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:34:05.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2024-09-15T21:34:05.770 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:34:05.770 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:34:05.770 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:34:05.770 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:34:05.770 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:05.770 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:05.771 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:34:05.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2024-09-15T21:34:05.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:34:05.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:34:05.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:34:05.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:34:05.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:34:05.991 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:34:05.991 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:34:05.991 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:34:05.991 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:34:05.991 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:34:05.992 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:05.992 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:05.992 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:34:05.992 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:34:05.992 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:34:06.107 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:34:06.107 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:34:06.108 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:34:06.108 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:34:06.108 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:34:06.109 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:34:06.109 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:34:06.109 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:34:06.109 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:34:06.109 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:06.109 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:06.109 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:34:06.109 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:34:06.109 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:34:06.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:34:06.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:06.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:34:06.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:34:06.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:34:06.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:34:06.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:34:07.111 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:34:07.111 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:34:07.112 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:34:07.112 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:34:07.112 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:07.112 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:07.112 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:34:07.113 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:34:07.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:34:07.148 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T21:34:07.148 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:34:07.148 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:34:07.148 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:34:07.148 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:07.148 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:07.149 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:34:07.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:34:07.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:34:07.247 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:34:07.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2024-09-15T21:34:07.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:07.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:34:07.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:34:07.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:34:07.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:34:07.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:34:07.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:34:07.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:34:07.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:34:07.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:34:07.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:34:07.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:34:07.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:34:07.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:34:07.263 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:34:07.263 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:34:07.263 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:34:07.265 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:34:07.265 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:07.265 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:07.265 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:34:07.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:34:07.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:34:07.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:34:07.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:34:07.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:34:07.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:34:07.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:34:07.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:34:07.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:34:07.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:34:07.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:34:07.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2024-09-15T21:34:07.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:34:07.267 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:34:07.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bf229594-d13e-4299-9fae-37f10279c329 2024-09-15T21:34:07.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 bf229594-d13e-4299-9fae-37f10279c329' 2024-09-15T21:34:07.267 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 bf229594-d13e-4299-9fae-37f10279c329 2024-09-15T21:34:07.267 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:34:07.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDPUudmqZr0EBAAi246aJGai5wiLo8UFU+sFw== 2024-09-15T21:34:07.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDPUudmqZr0EBAAi246aJGai5wiLo8UFU+sFw=="}' 2024-09-15T21:34:07.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bf229594-d13e-4299-9fae-37f10279c329 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:34:07.578 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:34:07.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:34:07.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQDPUudmqZr0EBAAi246aJGai5wiLo8UFU+sFw== --osd-uuid bf229594-d13e-4299-9fae-37f10279c329 2024-09-15T21:34:07.627 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:07.622+0000 7f7413f447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:07.645 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:07.642+0000 7f7413f447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:07.646 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:07.642+0000 7f7413f447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:07.646 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:07.642+0000 7f7413f447c0 -1 bdev(0x555a1e115c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:34:07.646 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:07.642+0000 7f7413f447c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:34:14.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:34:14.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:34:14.171 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:34:14.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:34:14.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:34:14.738 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:34:14.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:34:14.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2024-09-15T21:34:14.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:34:14.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:34:14.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:34:14.767 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:14.762+0000 7f697ce9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:14.790 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:14.786+0000 7f697ce9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:14.791 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:14.786+0000 7f697ce9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:15.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:34:15.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:34:15.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:34:15.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:34:15.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:34:15.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:15.221 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:34:15.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:34:15.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:15.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:15.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:16.595 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:16.590+0000 7f697ce9f7c0 -1 Falling back to public interface 2024-09-15T21:34:16.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:16.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:16.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:34:16.714 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:34:16.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:16.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:17.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:18.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:18.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:18.198 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:34:18.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:34:18.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:18.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:18.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:19.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:19.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:19.619 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:34:19.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:34:19.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:19.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:19.738+0000 7f697ce9f7c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:34:20.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:21.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:21.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:21.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:34:21.071 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:34:21.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:21.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:21.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:22.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:22.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:22.578 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:34:22.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:34:22.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:22.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:23.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:24.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:24.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:24.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:34:24.010 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:34:24.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:24.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:24.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:25.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:25.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:25.564 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:34:25.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:34:25.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:25.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:26.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:27.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:27.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:27.046 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:34:27.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:34:27.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:27.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:27.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:28.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:28.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:28.589 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:34:28.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:34:28.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:28.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:29.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:30.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:30.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:30.046 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:34:30.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:34:30.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:30.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:30.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:31.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:31.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:31.564 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:34:31.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:34:31.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:31.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:32.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:33.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:33.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:33.174 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:34:33.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:34:33.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:33.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:33.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:34.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:34.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:34.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:34:34.954 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:34:34.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:34.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:35.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:36.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:36.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:36.609 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:34:36.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:34:36.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:36.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:37.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:38.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:38.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:38.308 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:34:38.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:34:38.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:38.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:38.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:39.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:39.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:39.975 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:34:39.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:34:39.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:39.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:40.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:41.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:41.597 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:34:41.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:41.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:34:41.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:41.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:42.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:43.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:43.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:43.448 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:34:43.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:34:43.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:43.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:43.949 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1696666638,v1:127.0.0.1:6803/1696666638] [v2:127.0.0.1:6804/1696666638,v1:127.0.0.1:6805/1696666638] exists,up bf229594-d13e-4299-9fae-37f10279c329 2024-09-15T21:34:43.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:34:43.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:34:43.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:34:43.951 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T21:34:43.951 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:34:43.951 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:34:43.951 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:34:43.951 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:43.951 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:43.951 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-osd.0.asok 2024-09-15T21:34:43.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:34:43.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-osd.0.asok log flush 2024-09-15T21:34:44.055 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:34:44.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:34:44.072 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:34:05.802+0000 7fce0aed3e00 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T21:34:44.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-15T21:34:44.074 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:34:16.594+0000 7f697ce9f7c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T21:34:44.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T21:34:44.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:44.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:34:44.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:34:44.075 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:34:44.075 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:34:44.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:34:44.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:34:44.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:34:44.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:34:44.184 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:34:44.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:34:44.185 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:34:44.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:34:44.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:34:44.187 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:34:44.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:34:44.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:34:44.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:34:44.188 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:34:44.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:34:44.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:34:44.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:34:44.196 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:34:44.196 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:44.196 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:44.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:34:44.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:34:44.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:34:44.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T21:34:44.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T21:34:44.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:44.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-15T21:34:44.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:44.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:34:44.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:34:44.199 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:34:44.199 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:34:44.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:34:44.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:34:44.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:34:44.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:34:44.200 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:34:44.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:34:44.201 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:34:44.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:34:44.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:34:44.203 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:34:44.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:34:44.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:34:44.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:34:44.204 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:34:44.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:34:44.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:34:44.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:34:44.207 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:34:44.208 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:44.208 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:44.208 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:34:44.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:34:44.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:34:44.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T21:34:44.210 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:34:44.210 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:44.210 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:44.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21535 2024-09-15T21:34:44.212 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:34:44.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:34:44.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:34:44.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T21:34:44.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2024-09-15T21:34:44.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:44.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:34:44.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:34:44.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:34:44.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T21:34:44.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2024-09-15T21:34:44.691 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:34:44.692 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:34:44.692 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:34:44.692 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:34:44.692 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:44.692 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:44.693 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:34:44.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2024-09-15T21:34:44.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:34:44.968 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:34:44.968 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:34:44.968 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:34:44.968 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:34:44.969 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:34:44.969 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:34:44.969 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:34:44.970 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:34:44.970 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:34:44.970 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:44.970 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:44.970 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:34:44.970 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:34:44.970 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get fsid 2024-09-15T21:34:45.085 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:34:45.085 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:34:45.086 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:34:45.086 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:34:45.087 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:34:45.087 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:34:45.087 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:34:45.087 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:34:45.087 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:34:45.087 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:45.087 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:45.087 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:34:45.088 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:34:45.088 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21535/ceph-mon.a.asok config get mon_host 2024-09-15T21:34:45.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T21:34:45.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:45.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:34:45.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:34:45.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:34:45.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T21:34:45.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:34:46.098 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:34:46.098 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:34:46.099 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:34:46.099 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:34:46.099 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:46.099 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:46.099 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:34:46.100 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:34:46.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:34:46.134 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T21:34:46.134 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:34:46.134 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:34:46.134 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:34:46.134 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:46.134 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:46.134 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-mon.a.asok 2024-09-15T21:34:46.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:34:46.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-mon.a.asok log flush 2024-09-15T21:34:46.232 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:34:46.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2024-09-15T21:34:46.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T21:34:46.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:34:46.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:34:46.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:34:46.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T21:34:46.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T21:34:46.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:34:46.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:46.249 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:34:46.251 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:34:46.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' 2024-09-15T21:34:46.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:34:46.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:34:46.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:34:46.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T21:34:46.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T21:34:46.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:34:46.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:34:46.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:34:46.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:34:46.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:34:46.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2024-09-15T21:34:46.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T21:34:46.252 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:34:46.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d30ef3ba-af86-4000-8316-efa6db0c8865 2024-09-15T21:34:46.252 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 d30ef3ba-af86-4000-8316-efa6db0c8865 2024-09-15T21:34:46.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d30ef3ba-af86-4000-8316-efa6db0c8865' 2024-09-15T21:34:46.253 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:34:46.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD2UudmqPoFEBAA5HSgnHb2t0+NpNLEx5eyyw== 2024-09-15T21:34:46.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD2UudmqPoFEBAA5HSgnHb2t0+NpNLEx5eyyw=="}' 2024-09-15T21:34:46.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d30ef3ba-af86-4000-8316-efa6db0c8865 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T21:34:46.556 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:34:46.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T21:34:46.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQD2UudmqPoFEBAA5HSgnHb2t0+NpNLEx5eyyw== --osd-uuid d30ef3ba-af86-4000-8316-efa6db0c8865 2024-09-15T21:34:46.604 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:46.597+0000 7f7020ca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:46.616 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:46.609+0000 7f7020ca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:46.617 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:46.613+0000 7f7020ca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:46.618 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:46.613+0000 7f7020ca47c0 -1 bdev(0x55bdffd15c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:34:46.618 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:46.613+0000 7f7020ca47c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T21:34:52.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T21:34:52.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:34:52.818 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:34:52.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:34:52.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:34:53.403 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:34:53.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:34:53.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:34:53.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=893c07ef-6324-47dc-86ed-f4dff41aca4c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.21535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2024-09-15T21:34:53.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:34:53.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:34:53.436 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:53.429+0000 7f61063d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:53.463 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:53.457+0000 7f61063d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:53.464 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:53.461+0000 7f61063d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:53.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:34:53.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:34:53.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:34:53.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:34:53.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:34:53.837 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:34:53.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:53.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:34:53.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:53.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:54.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:54.963 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:54.957+0000 7f61063d47c0 -1 Falling back to public interface 2024-09-15T21:34:55.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:55.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:55.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:34:55.263 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:34:55.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:55.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:55.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:56.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:56.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:56.694 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:34:56.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:34:56.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:56.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:57.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:57.746 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:34:57.741+0000 7f61063d47c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:34:58.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:58.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:58.121 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:34:58.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:34:58.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:58.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:58.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:59.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:59.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:59.564 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:34:59.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:34:59.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:59.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:00.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:01.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:01.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:01.004 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:35:01.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:35:01.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:01.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:01.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:02.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:02.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:02.577 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:35:02.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:35:02.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:02.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:03.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:04.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:04.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:04.010 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:35:04.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:35:04.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:04.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:04.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:05.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:05.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:05.557 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:35:05.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:35:05.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:05.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:06.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:07.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:07.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:07.023 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:35:07.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:35:07.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:07.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:07.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:08.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:08.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:08.581 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:35:08.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:35:08.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:08.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:09.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:10.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:10.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:10.037 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:35:10.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:35:10.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:10.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:10.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:11.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:11.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:11.710 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:35:11.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:35:11.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:11.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:12.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:13.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:13.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:13.434 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:35:13.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:35:13.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:13.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:14.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:15.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:15.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:15.449 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:35:15.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:35:15.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:15.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:16.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:17.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:17.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:17.389 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:35:17.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:35:17.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:17.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:18.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:19.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:19.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:19.440 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:35:19.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:35:19.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:19.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:20.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:21.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:21.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:21.301 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:35:21.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:35:21.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:21.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:21.790 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2768760770,v1:127.0.0.1:6803/2768760770] [v2:127.0.0.1:6804/2768760770,v1:127.0.0.1:6805/2768760770] exists,up d30ef3ba-af86-4000-8316-efa6db0c8865 2024-09-15T21:35:21.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:35:21.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:35:21.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:35:21.791 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T21:35:21.791 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:35:21.792 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:35:21.792 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:35:21.792 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:21.792 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:35:21.792 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21535/ceph-osd.0.asok 2024-09-15T21:35:21.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T21:35:21.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.21535/ceph-osd.0.asok log flush 2024-09-15T21:35:21.888 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:35:21.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-15T21:35:21.903 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:34:44.725+0000 7fea0d0b0e00 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T21:35:21.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-15T21:35:21.905 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:34:54.961+0000 7f61063d47c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T21:35:21.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T21:35:21.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:35:21.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:35:21.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:35:21.907 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:35:21.907 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:35:21.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:35:21.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:35:21.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:35:22.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:35:22.217 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:35:22.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:35:22.219 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:35:22.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:35:22.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:35:22.220 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:35:22.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:35:22.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:35:22.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:35:22.222 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:35:22.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:35:22.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:35:22.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:35:22.231 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:35:22.231 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:22.231 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:35:22.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:35:22.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:35:22.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:35:22.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2024-09-15T21:35:22.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T21:35:22.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/test-erasure-code-plugins 0 2024-09-15T21:35:22.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T21:35:22.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T21:35:22.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T21:35:22.233 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:35:22.234 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:35:22.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:35:22.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:35:22.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:35:22.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:35:22.236 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:35:22.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:35:22.237 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:35:22.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:35:22.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:35:22.239 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:35:22.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:35:22.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:35:22.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:35:22.241 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:35:22.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:35:22.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T21:35:22.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T21:35:22.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:35:22.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:22.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21535 2024-09-15T21:35:22.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21535 2024-09-15T21:35:22.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:35:22.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:35:22.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T21:35:22.246 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T21:35:22.247 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T21:35:22.257 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2024-09-15T21:35:22.257 DEBUG:teuthology.orchestra.run.smithi104:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2024-09-15T21:35:22.311 INFO:tasks.workunit.client.0.smithi104.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T21:35:22.319 INFO:tasks.workunit.client.0.smithi104.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T21:35:22.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T21:35:22.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T21:35:22.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T21:35:22.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T21:35:22.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T21:35:22.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T21:35:22.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T21:35:22.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T21:35:22.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/test-erasure-code 2024-09-15T21:35:22.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2024-09-15T21:35:22.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none ' 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:35:22.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2024-09-15T21:35:22.322 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:35:22.322 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:35:22.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:35:22.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:35:22.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:35:22.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:35:22.323 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:35:22.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:35:22.323 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:35:22.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:35:22.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:35:22.324 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:35:22.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:35:22.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:35:22.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:35:22.325 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:35:22.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:35:22.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:35:22.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2024-09-15T21:35:22.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:35:22.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:22.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:35:22.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.50061 2024-09-15T21:35:22.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:35:22.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:35:22.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code 2024-09-15T21:35:22.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:35:22.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:22.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:35:22.331 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.50061 2024-09-15T21:35:22.332 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:35:22.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T21:35:22.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T21:35:22.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:35:22.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2024-09-15T21:35:22.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2024-09-15T21:35:22.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code 2024-09-15T21:35:22.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:35:22.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:35:22.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:35:22.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code/a 2024-09-15T21:35:22.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2024-09-15T21:35:22.803 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:35:22.803 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:35:22.804 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:35:22.804 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:35:22.804 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:22.805 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:35:22.805 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:35:22.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:35:23.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:35:23.049 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:35:23.049 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:35:23.049 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:35:23.049 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:35:23.050 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:35:23.050 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:35:23.050 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:35:23.050 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:35:23.051 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:35:23.051 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:23.051 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:35:23.052 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.50061/ceph-mon.a.asok 2024-09-15T21:35:23.052 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:35:23.052 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.50061/ceph-mon.a.asok config get fsid 2024-09-15T21:35:23.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:35:23.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:35:23.171 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:35:23.171 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:35:23.171 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:35:23.171 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:35:23.171 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:35:23.171 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:35:23.171 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:35:23.171 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:23.172 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:35:23.172 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.50061/ceph-mon.a.asok 2024-09-15T21:35:23.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:35:23.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.50061/ceph-mon.a.asok config get mon_host 2024-09-15T21:35:23.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2024-09-15T21:35:23.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code 2024-09-15T21:35:23.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:35:23.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:35:23.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:35:23.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code/x 2024-09-15T21:35:23.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:35:24.212 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:35:24.212 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:35:24.212 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:35:24.212 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:35:24.212 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:24.212 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:35:24.212 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:35:24.213 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:35:24.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:35:24.247 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2024-09-15T21:35:24.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:35:24.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:35:24.248 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:35:24.248 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:24.248 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:35:24.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.50061/ceph-mon.a.asok 2024-09-15T21:35:24.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2024-09-15T21:35:24.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.50061/ceph-mon.a.asok log flush 2024-09-15T21:35:24.347 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:35:24.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2024-09-15T21:35:24.364 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:35:22.841+0000 7f02c4b45e00 0 load: jerasure load: lrc 2024-09-15T21:35:24.365 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2024-09-15T21:35:24.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T21:35:24.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2024-09-15T21:35:24.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-15T21:35:24.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:35:24.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/0 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:35:24.368 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:35:24.369 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:35:24.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:35:24.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:35:24.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:35:24.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:35:24.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:35:24.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/0 2024-09-15T21:35:24.371 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:35:24.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4362ab40-3f87-466d-8e31-3ffa7c5f7cc1 2024-09-15T21:35:24.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4362ab40-3f87-466d-8e31-3ffa7c5f7cc1' 2024-09-15T21:35:24.372 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 4362ab40-3f87-466d-8e31-3ffa7c5f7cc1 2024-09-15T21:35:24.372 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:35:24.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAcU+dmfiFTFxAANOD34LFP/d6SdmZio+gIsQ== 2024-09-15T21:35:24.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAcU+dmfiFTFxAANOD34LFP/d6SdmZio+gIsQ=="}' 2024-09-15T21:35:24.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4362ab40-3f87-466d-8e31-3ffa7c5f7cc1 -i td/test-erasure-code/0/new.json 2024-09-15T21:35:24.709 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:35:24.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/0/new.json 2024-09-15T21:35:24.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAcU+dmfiFTFxAANOD34LFP/d6SdmZio+gIsQ== --osd-uuid 4362ab40-3f87-466d-8e31-3ffa7c5f7cc1 2024-09-15T21:35:24.767 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:35:24.761+0000 7f23703497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:24.786 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:35:24.781+0000 7f23703497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:24.787 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:35:24.781+0000 7f23703497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:24.788 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:35:24.781+0000 7f23703497c0 -1 bdev(0x5579a3095c00 td/test-erasure-code/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:35:24.788 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:35:24.781+0000 7f23703497c0 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2024-09-15T21:35:30.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/0/keyring 2024-09-15T21:35:30.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:35:30.976 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:35:30.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:35:30.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:35:31.556 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:35:31.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:35:31.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:35:31.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:35:31.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:35:31.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:35:31.587 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:35:31.581+0000 7f8de48ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:31.590 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:35:31.585+0000 7f8de48ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:31.591 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:35:31.585+0000 7f8de48ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:32.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:35:32.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:35:32.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:35:32.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:35:32.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:35:32.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:32.002 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:35:32.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:35:32.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:32.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:32.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:32.639 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:35:32.633+0000 7f8de48ec7c0 -1 Falling back to public interface 2024-09-15T21:35:33.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:33.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:33.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:35:33.436 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:35:33.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:33.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:33.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:34.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:34.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:34.852 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:35:34.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:35:34.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:34.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:35.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:35:35.485+0000 7f8de48ec7c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:35:36.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:36.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:36.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:35:36.282 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:35:36.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:36.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:36.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:37.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:37.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:37.745 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:35:37.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:35:37.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:37.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:38.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:39.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:39.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:39.177 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:35:39.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:35:39.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:39.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:39.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:40.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:40.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:40.622 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:35:40.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:35:40.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:40.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:41.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:42.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:42.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:42.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:35:42.057 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:35:42.058 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:42.058 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:42.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:43.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:43.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:43.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:35:43.498 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:35:43.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:43.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:43.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:44.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:44.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:44.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:35:44.938 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:35:44.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:44.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:45.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:46.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:46.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:46.511 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:35:46.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:35:46.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:46.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:46.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:47.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:47.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:47.957 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:35:47.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:35:47.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:47.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:49.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:50.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:50.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:50.155 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:35:50.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:35:50.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:50.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:50.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:51.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:51.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:51.908 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:35:51.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:35:51.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:51.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:52.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:53.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:53.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:53.756 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:35:53.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:35:53.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:53.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:54.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:55.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:55.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:55.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:35:55.905 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:35:55.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:55.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:56.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:57.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:57.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:57.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:35:57.800 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:35:57.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:57.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:58.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:59.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:59.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:59.522 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:35:59.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:35:59.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:59.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:36:00.023 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/204850398,v1:127.0.0.1:6803/204850398] [v2:127.0.0.1:6804/204850398,v1:127.0.0.1:6805/204850398] exists,up 4362ab40-3f87-466d-8e31-3ffa7c5f7cc1 2024-09-15T21:36:00.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:36:00.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:36:00.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:36:00.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T21:36:00.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2024-09-15T21:36:00.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-15T21:36:00.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:36:00.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/1 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:36:00.028 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:36:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:36:00.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:36:00.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:36:00.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:36:00.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/1 2024-09-15T21:36:00.030 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:36:00.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3d1504db-b72a-477e-a8d1-a207478773b9 2024-09-15T21:36:00.030 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 3d1504db-b72a-477e-a8d1-a207478773b9 2024-09-15T21:36:00.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3d1504db-b72a-477e-a8d1-a207478773b9' 2024-09-15T21:36:00.031 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:36:00.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBAU+dmUOfBAhAAVKdD0w4so+SRJIA3SyDIWg== 2024-09-15T21:36:00.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBAU+dmUOfBAhAAVKdD0w4so+SRJIA3SyDIWg=="}' 2024-09-15T21:36:00.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3d1504db-b72a-477e-a8d1-a207478773b9 -i td/test-erasure-code/1/new.json 2024-09-15T21:36:00.546 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:36:00.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/1/new.json 2024-09-15T21:36:00.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAU+dmUOfBAhAAVKdD0w4so+SRJIA3SyDIWg== --osd-uuid 3d1504db-b72a-477e-a8d1-a207478773b9 2024-09-15T21:36:00.596 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:00.593+0000 7f8cde95e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:00.658 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:00.653+0000 7f8cde95e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:00.659 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:00.653+0000 7f8cde95e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:00.660 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:00.653+0000 7f8cde95e7c0 -1 bdev(0x55bc43b3bc00 td/test-erasure-code/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:36:00.660 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:00.657+0000 7f8cde95e7c0 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2024-09-15T21:36:07.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/1/keyring 2024-09-15T21:36:07.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:36:07.368 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T21:36:07.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:36:07.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:36:07.948 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T21:36:07.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:36:07.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:36:07.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:36:07.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:36:07.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:36:07.973 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:07.969+0000 7f4b88cd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:07.976 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:07.969+0000 7f4b88cd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:07.977 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:07.973+0000 7f4b88cd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:08.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:36:08.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:36:08.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:36:08.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:36:08.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:36:08.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:08.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:36:08.412 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:36:08.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:08.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:08.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:09.295 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:09.289+0000 7f4b88cd17c0 -1 Falling back to public interface 2024-09-15T21:36:09.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:09.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:09.902 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:36:09.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:36:09.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:09.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:10.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:11.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:11.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:11.362 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:36:11.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:36:11.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:11.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:11.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:12.282 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:12.277+0000 7f4b88cd17c0 -1 osd.1 0 log_to_monitors true 2024-09-15T21:36:12.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:12.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:12.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:36:12.784 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:36:12.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:12.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:13.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:14.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:14.603 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:36:14.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:14.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:36:14.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:14.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:15.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:16.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:16.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:16.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:36:16.255 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:36:16.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:16.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:16.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:17.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:17.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:17.739 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:36:17.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:36:17.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:17.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:18.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:19.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:19.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:19.276 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:36:19.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:36:19.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:19.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:19.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:20.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:20.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:20.776 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:36:20.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:36:20.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:20.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:21.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:22.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:22.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:22.361 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:36:22.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:36:22.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:22.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:22.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:23.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:23.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:23.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:36:23.808 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:36:23.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:23.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:24.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:25.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:25.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:25.351 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:36:25.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:36:25.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:25.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:25.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:26.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:26.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:26.836 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:36:26.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:36:26.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:26.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:28.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:29.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:29.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:29.008 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:36:29.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:36:29.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:29.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:30.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:31.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:31.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:31.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:36:31.142 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:36:31.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:31.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:31.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:32.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:32.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:32.637 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:36:32.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:36:32.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:32.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:33.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:34.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:34.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:34.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:36:34.417 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:36:34.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:34.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:35.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:36:36.038 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:36:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:36.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:37.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:37.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:37.752 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:36:37.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:36:37.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:37.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:38.260 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2379193483,v1:127.0.0.1:6811/2379193483] [v2:127.0.0.1:6812/2379193483,v1:127.0.0.1:6813/2379193483] exists,up 3d1504db-b72a-477e-a8d1-a207478773b9 2024-09-15T21:36:38.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:36:38.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:36:38.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:36:38.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T21:36:38.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2024-09-15T21:36:38.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/2 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:36:38.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:36:38.264 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:36:38.264 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:36:38.264 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:36:38.264 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:36:38.264 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:36:38.264 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:36:38.264 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:36:38.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:36:38.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:36:38.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/2 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:36:38.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=38fc7753-4b92-4deb-acb8-52856e9b8697 2024-09-15T21:36:38.267 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 38fc7753-4b92-4deb-acb8-52856e9b8697 2024-09-15T21:36:38.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 38fc7753-4b92-4deb-acb8-52856e9b8697' 2024-09-15T21:36:38.267 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:36:38.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBmU+dm5unDEBAAmyDfP+HuUtyhbm9YVsAtww== 2024-09-15T21:36:38.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBmU+dm5unDEBAAmyDfP+HuUtyhbm9YVsAtww=="}' 2024-09-15T21:36:38.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 38fc7753-4b92-4deb-acb8-52856e9b8697 -i td/test-erasure-code/2/new.json 2024-09-15T21:36:38.796 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:36:38.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/2/new.json 2024-09-15T21:36:38.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBmU+dm5unDEBAAmyDfP+HuUtyhbm9YVsAtww== --osd-uuid 38fc7753-4b92-4deb-acb8-52856e9b8697 2024-09-15T21:36:38.848 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:38.840+0000 7f8272a277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:38.853 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:38.848+0000 7f8272a277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:38.854 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:38.848+0000 7f8272a277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:38.854 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:38.848+0000 7f8272a277c0 -1 bdev(0x55fec46fbc00 td/test-erasure-code/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:36:38.855 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:38.848+0000 7f8272a277c0 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2024-09-15T21:36:44.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/2/keyring 2024-09-15T21:36:44.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:36:44.994 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T21:36:44.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:36:44.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:36:45.608 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T21:36:45.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:36:45.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:36:45.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:36:45.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:36:45.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:36:45.639 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:45.632+0000 7f051e6867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:45.641 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:45.636+0000 7f051e6867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:45.642 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:45.636+0000 7f051e6867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:46.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:36:46.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:36:46.087 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:36:46.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:36:46.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:36:46.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:36:46.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:46.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:36:46.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:46.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:46.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:46.619 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:46.612+0000 7f051e6867c0 -1 Falling back to public interface 2024-09-15T21:36:47.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:47.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:47.563 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:36:47.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:36:47.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:47.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:48.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:49.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:49.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:49.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:36:49.040 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:36:49.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:49.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:49.470 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:36:49.464+0000 7f051e6867c0 -1 osd.2 0 log_to_monitors true 2024-09-15T21:36:49.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:50.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:50.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:50.538 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:36:50.539 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:36:50.539 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:50.539 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:50.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:51.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:51.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:51.993 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:36:51.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:36:51.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:51.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:52.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:53.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:53.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:53.488 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:36:53.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:36:53.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:53.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:54.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:55.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:55.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:55.029 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:36:55.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:36:55.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:55.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:55.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:56.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:56.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:56.583 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:36:56.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:36:56.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:56.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:57.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:58.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:58.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:58.020 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:36:58.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:36:58.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:58.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:58.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:59.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:59.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:59.529 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:36:59.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:36:59.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:59.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:59.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:00.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:00.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:00.977 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:37:00.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:37:00.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:00.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:01.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:02.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:02.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:02.474 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:37:02.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:37:02.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:02.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:02.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:03.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:03.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:03.910 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:37:03.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:37:03.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:03.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:04.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:05.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:05.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:05.714 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:37:05.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:37:05.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:05.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:06.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:07.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:07.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:07.867 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:37:07.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:37:07.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:07.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:08.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:09.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:09.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:09.940 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:37:09.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:37:09.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:09.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:10.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:11.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:11.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:11.853 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:37:11.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:37:11.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:11.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:12.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:13.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:13.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:13.508 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:37:13.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:37:13.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:13.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:14.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:15.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:15.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:15.629 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:37:15.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:37:15.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:15.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:16.058 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3349409135,v1:127.0.0.1:6819/3349409135] [v2:127.0.0.1:6820/3349409135,v1:127.0.0.1:6821/3349409135] exists,up 38fc7753-4b92-4deb-acb8-52856e9b8697 2024-09-15T21:37:16.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:37:16.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:37:16.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:37:16.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T21:37:16.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2024-09-15T21:37:16.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-15T21:37:16.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:37:16.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T21:37:16.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:37:16.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/3 2024-09-15T21:37:16.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:37:16.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:37:16.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:37:16.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:37:16.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-15T21:37:16.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-15T21:37:16.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:37:16.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:37:16.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:37:16.063 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:37:16.063 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:37:16.063 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:37:16.063 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:37:16.063 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:37:16.063 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:37:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/3 2024-09-15T21:37:16.066 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:37:16.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7e795b35-bf62-4700-b20b-41066d193de7 2024-09-15T21:37:16.066 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 7e795b35-bf62-4700-b20b-41066d193de7 2024-09-15T21:37:16.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 7e795b35-bf62-4700-b20b-41066d193de7' 2024-09-15T21:37:16.066 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:37:16.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCMU+dmucC6BBAAdCFiPc16f6NC0Poa2TbvGg== 2024-09-15T21:37:16.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCMU+dmucC6BBAAdCFiPc16f6NC0Poa2TbvGg=="}' 2024-09-15T21:37:16.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7e795b35-bf62-4700-b20b-41066d193de7 -i td/test-erasure-code/3/new.json 2024-09-15T21:37:16.687 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:37:16.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/3/new.json 2024-09-15T21:37:16.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMU+dmucC6BBAAdCFiPc16f6NC0Poa2TbvGg== --osd-uuid 7e795b35-bf62-4700-b20b-41066d193de7 2024-09-15T21:37:16.737 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:16.732+0000 7f6a4910d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:16.745 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:16.740+0000 7f6a4910d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:16.747 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:16.740+0000 7f6a4910d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:16.747 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:16.740+0000 7f6a4910d7c0 -1 bdev(0x55fc15c7fc00 td/test-erasure-code/3/block) open stat got: (1) Operation not permitted 2024-09-15T21:37:16.747 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:16.740+0000 7f6a4910d7c0 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2024-09-15T21:37:23.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/3/keyring 2024-09-15T21:37:23.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:37:23.195 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T21:37:23.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T21:37:23.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:37:23.791 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T21:37:23.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T21:37:23.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:37:23.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:37:23.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:37:23.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:37:23.822 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:23.816+0000 7f91fdd5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:23.850 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:23.844+0000 7f91fdd5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:23.852 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:23.848+0000 7f91fdd5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:24.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T21:37:24.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:37:24.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:37:24.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:37:24.286 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:37:24.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:37:24.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:24.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:37:24.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:24.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:24.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:25.343 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:25.336+0000 7f91fdd5d7c0 -1 Falling back to public interface 2024-09-15T21:37:25.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:25.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:25.721 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:37:25.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:37:25.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:25.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:26.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:27.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:27.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:27.161 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:37:27.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:37:27.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:27.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:27.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:28.268 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:28.260+0000 7f91fdd5d7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T21:37:28.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:28.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:28.670 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:37:28.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:37:28.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:28.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:29.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:30.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:30.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:30.149 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:37:30.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:37:30.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:30.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:30.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:31.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:31.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:31.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:37:31.579 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:37:31.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:31.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:32.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:33.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:33.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:33.119 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:37:33.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:37:33.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:33.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:33.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:34.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:34.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:34.706 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:37:34.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:37:34.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:34.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:35.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:36.239 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:37:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:37:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:36.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:37.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:37.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:37.705 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:37:37.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:37:37.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:37.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:38.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:39.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:39.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:39.231 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:37:39.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:37:39.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:39.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:39.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:40.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:40.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:40.775 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:37:40.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:37:40.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:40.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:41.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:42.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:42.236 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:37:42.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:42.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:37:42.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:42.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:42.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:43.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:43.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:43.792 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:37:43.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:37:43.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:43.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:44.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:45.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:45.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:45.615 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:37:45.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:37:45.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:45.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:46.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:47.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:47.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:47.706 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:37:47.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:37:47.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:47.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:48.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:49.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:49.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:49.617 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:37:49.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:37:49.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:49.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:50.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:51.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:51.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:51.619 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:37:51.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:37:51.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:51.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:52.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:53.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:53.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:53.429 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:37:53.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:37:53.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:53.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:37:53.884 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/839845376,v1:127.0.0.1:6827/839845376] [v2:127.0.0.1:6828/839845376,v1:127.0.0.1:6829/839845376] exists,up 7e795b35-bf62-4700-b20b-41066d193de7 2024-09-15T21:37:53.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:37:53.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:37:53.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:37:53.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T21:37:53.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2024-09-15T21:37:53.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-15T21:37:53.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:37:53.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-15T21:37:53.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/4 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:37:53.888 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:37:53.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:37:53.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:37:53.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:37:53.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/4 2024-09-15T21:37:53.890 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:37:53.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ede04d59-2648-4bdf-bfc2-78e688ed4a5b 2024-09-15T21:37:53.891 INFO:tasks.workunit.client.0.smithi104.stdout:add osd4 ede04d59-2648-4bdf-bfc2-78e688ed4a5b 2024-09-15T21:37:53.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 ede04d59-2648-4bdf-bfc2-78e688ed4a5b' 2024-09-15T21:37:53.891 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:37:53.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCxU+dmKEEENhAA4sDzlxevq/KnYvwVFjvQPg== 2024-09-15T21:37:53.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCxU+dmKEEENhAA4sDzlxevq/KnYvwVFjvQPg=="}' 2024-09-15T21:37:53.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ede04d59-2648-4bdf-bfc2-78e688ed4a5b -i td/test-erasure-code/4/new.json 2024-09-15T21:37:54.409 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:37:54.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/4/new.json 2024-09-15T21:37:54.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxU+dmKEEENhAA4sDzlxevq/KnYvwVFjvQPg== --osd-uuid ede04d59-2648-4bdf-bfc2-78e688ed4a5b 2024-09-15T21:37:54.453 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:54.448+0000 7f4fba79f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:54.471 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:54.468+0000 7f4fba79f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:54.473 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:54.468+0000 7f4fba79f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:54.473 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:54.468+0000 7f4fba79f7c0 -1 bdev(0x55a125945c00 td/test-erasure-code/4/block) open stat got: (1) Operation not permitted 2024-09-15T21:37:54.473 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:37:54.468+0000 7f4fba79f7c0 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2024-09-15T21:38:01.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/4/keyring 2024-09-15T21:38:01.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:38:01.266 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd4 key to auth repository 2024-09-15T21:38:01.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-15T21:38:01.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:38:01.906 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.4 2024-09-15T21:38:01.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-15T21:38:01.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:38:01.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:38:01.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:38:01.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:38:01.937 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:01.936+0000 7fc60406e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:01.951 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:01.948+0000 7fc60406e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:01.952 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:01.952+0000 7fc60406e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:02.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-15T21:38:02.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:38:02.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-15T21:38:02.387 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:38:02.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:38:02.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:38:02.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:02.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:38:02.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:02.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:02.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:03.023 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:03.020+0000 7fc60406e7c0 -1 Falling back to public interface 2024-09-15T21:38:03.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:03.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:03.819 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:38:03.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:38:03.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:03.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:04.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:05.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:05.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:05.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:38:05.237 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:38:05.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:05.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:05.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:06.006 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:06.004+0000 7fc60406e7c0 -1 osd.4 0 log_to_monitors true 2024-09-15T21:38:06.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:06.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:06.688 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:38:06.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:38:06.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:06.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:07.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:08.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:08.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:08.276 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:38:08.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:38:08.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:08.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:08.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:09.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:09.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:09.861 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:38:09.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:38:09.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:09.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:10.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:11.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:11.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:11.297 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:38:11.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:38:11.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:11.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:11.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:12.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:12.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:12.809 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:38:12.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:38:12.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:12.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:13.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:14.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:14.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:14.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:38:14.352 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:38:14.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:14.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:14.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:15.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:15.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:15.867 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:38:15.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:38:15.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:15.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:16.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:17.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:17.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:17.327 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:38:17.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:38:17.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:17.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:17.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:18.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:18.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:18.749 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:38:18.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:38:18.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:18.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:19.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:20.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:20.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:20.224 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:38:20.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:38:20.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:20.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:21.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:22.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:22.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:22.377 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:38:22.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:38:22.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:22.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:23.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:24.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:24.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:24.098 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:38:24.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:38:24.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:24.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:24.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:25.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:25.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:25.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:38:25.848 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:38:25.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:25.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:26.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:27.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:27.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:27.887 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:38:27.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:38:27.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:27.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:28.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:29.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:29.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:29.662 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:38:29.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:38:29.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:29.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:30.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:31.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:31.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:31.271 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:38:31.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:38:31.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:31.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:38:31.708 INFO:tasks.workunit.client.0.smithi104.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/588515789,v1:127.0.0.1:6835/588515789] [v2:127.0.0.1:6836/588515789,v1:127.0.0.1:6837/588515789] exists,up ede04d59-2648-4bdf-bfc2-78e688ed4a5b 2024-09-15T21:38:31.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:38:31.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:38:31.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/5 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:38:31.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-15T21:38:31.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-15T21:38:31.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:38:31.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:38:31.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:38:31.712 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:38:31.712 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:38:31.712 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:38:31.712 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:38:31.712 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:38:31.713 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:38:31.713 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:38:31.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:38:31.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:38:31.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:38:31.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:38:31.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:38:31.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:38:31.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:38:31.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:38:31.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:38:31.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:38:31.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:38:31.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:38:31.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/5 2024-09-15T21:38:31.715 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:38:31.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a5825e7b-74c1-47c2-abb3-f77a8dbdcad5 2024-09-15T21:38:31.715 INFO:tasks.workunit.client.0.smithi104.stdout:add osd5 a5825e7b-74c1-47c2-abb3-f77a8dbdcad5 2024-09-15T21:38:31.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 a5825e7b-74c1-47c2-abb3-f77a8dbdcad5' 2024-09-15T21:38:31.716 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:38:31.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDXU+dmMeuCKxAAR+eUVRxHfIPV0ZPITxtueA== 2024-09-15T21:38:31.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDXU+dmMeuCKxAAR+eUVRxHfIPV0ZPITxtueA=="}' 2024-09-15T21:38:31.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a5825e7b-74c1-47c2-abb3-f77a8dbdcad5 -i td/test-erasure-code/5/new.json 2024-09-15T21:38:32.268 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:38:32.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/5/new.json 2024-09-15T21:38:32.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXU+dmMeuCKxAAR+eUVRxHfIPV0ZPITxtueA== --osd-uuid a5825e7b-74c1-47c2-abb3-f77a8dbdcad5 2024-09-15T21:38:32.317 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:32.315+0000 7fa8f91a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:32.408 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:32.407+0000 7fa8f91a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:32.410 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:32.407+0000 7fa8f91a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:32.411 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:32.407+0000 7fa8f91a27c0 -1 bdev(0x55e8fe265c00 td/test-erasure-code/5/block) open stat got: (1) Operation not permitted 2024-09-15T21:38:32.411 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:32.407+0000 7fa8f91a27c0 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2024-09-15T21:38:39.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/5/keyring 2024-09-15T21:38:39.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:38:39.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-15T21:38:39.190 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd5 key to auth repository 2024-09-15T21:38:39.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:38:39.755 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.5 2024-09-15T21:38:39.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-15T21:38:39.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:38:39.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:38:39.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:38:39.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:38:39.786 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:39.783+0000 7f726ee7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:39.812 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:39.807+0000 7f726ee7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:39.813 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:39.811+0000 7f726ee7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:40.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-15T21:38:40.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:38:40.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:38:40.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:38:40.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:38:40.241 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:38:40.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:40.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:38:40.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:40.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:40.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:40.879 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:40.875+0000 7f726ee7e7c0 -1 Falling back to public interface 2024-09-15T21:38:41.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:41.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:41.734 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:38:41.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:38:41.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:41.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:42.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:43.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:43.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:43.175 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:38:43.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:38:43.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:43.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:43.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:43.824 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:38:43.823+0000 7f726ee7e7c0 -1 osd.5 0 log_to_monitors true 2024-09-15T21:38:44.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:44.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:44.760 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:38:44.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:38:44.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:44.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:45.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:46.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:46.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:46.312 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:38:46.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:38:46.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:46.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:46.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:47.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:47.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:47.921 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:38:47.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:38:47.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:47.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:48.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:49.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:49.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:49.393 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:38:49.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:38:49.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:49.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:50.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:51.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:51.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:51.029 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:38:51.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:38:51.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:51.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:51.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:52.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:52.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:52.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:38:52.650 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:38:52.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:52.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:53.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:54.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:54.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:54.119 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:38:54.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:38:54.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:54.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:54.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:55.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:55.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:55.543 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:38:55.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:38:55.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:55.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:56.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:57.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:57.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:57.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:38:57.014 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:38:57.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:57.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:57.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:58.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:58.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:58.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:38:58.522 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:38:58.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:58.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:38:59.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:00.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:00.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:00.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:39:00.387 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:39:00.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:00.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:39:01.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:02.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:02.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:02.217 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:39:02.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:39:02.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:02.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:39:02.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:03.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:03.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:03.721 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:39:03.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:39:03.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:03.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:39:04.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:05.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:05.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:05.631 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:39:05.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:39:05.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:05.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:39:06.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:07.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:07.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:07.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:39:07.665 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:39:07.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:07.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:39:08.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:09.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:09.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:09.512 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:39:09.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:39:09.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:09.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:39:09.942 INFO:tasks.workunit.client.0.smithi104.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2432624105,v1:127.0.0.1:6843/2432624105] [v2:127.0.0.1:6844/2432624105,v1:127.0.0.1:6845/2432624105] exists,up a5825e7b-74c1-47c2-abb3-f77a8dbdcad5 2024-09-15T21:39:09.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:39:09.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/6 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:39:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-15T21:39:09.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-15T21:39:09.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:39:09.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:39:09.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:39:09.947 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:39:09.947 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:39:09.947 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:39:09.947 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:39:09.947 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:39:09.948 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:39:09.948 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:39:09.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:39:09.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:39:09.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:39:09.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:39:09.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:39:09.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:39:09.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:39:09.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:39:09.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:39:09.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:39:09.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:39:09.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:39:09.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/6 2024-09-15T21:39:09.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:39:09.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8f04fbd0-c08f-4b32-bb5e-b3d10aa3bc9b 2024-09-15T21:39:09.950 INFO:tasks.workunit.client.0.smithi104.stdout:add osd6 8f04fbd0-c08f-4b32-bb5e-b3d10aa3bc9b 2024-09-15T21:39:09.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 8f04fbd0-c08f-4b32-bb5e-b3d10aa3bc9b' 2024-09-15T21:39:09.950 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:39:09.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD9U+dmZKfDORAAW80+d0FyuMmXp5+U8GPMLw== 2024-09-15T21:39:09.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD9U+dmZKfDORAAW80+d0FyuMmXp5+U8GPMLw=="}' 2024-09-15T21:39:09.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8f04fbd0-c08f-4b32-bb5e-b3d10aa3bc9b -i td/test-erasure-code/6/new.json 2024-09-15T21:39:10.478 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:39:10.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/6/new.json 2024-09-15T21:39:10.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9U+dmZKfDORAAW80+d0FyuMmXp5+U8GPMLw== --osd-uuid 8f04fbd0-c08f-4b32-bb5e-b3d10aa3bc9b 2024-09-15T21:39:10.530 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:10.527+0000 7f179e4f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:10.533 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:10.531+0000 7f179e4f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:10.534 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:10.531+0000 7f179e4f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:10.534 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:10.531+0000 7f179e4f37c0 -1 bdev(0x561f9fdcbc00 td/test-erasure-code/6/block) open stat got: (1) Operation not permitted 2024-09-15T21:39:10.534 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:10.531+0000 7f179e4f37c0 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2024-09-15T21:39:16.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/6/keyring 2024-09-15T21:39:16.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:39:16.937 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd6 key to auth repository 2024-09-15T21:39:16.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-15T21:39:16.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:39:17.568 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.6 2024-09-15T21:39:17.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-15T21:39:17.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:39:17.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:39:17.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:39:17.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:39:17.599 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:17.595+0000 7f539762a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:17.628 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:17.623+0000 7f539762a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:17.628 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:17.627+0000 7f539762a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:18.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-15T21:39:18.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:39:18.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-15T21:39:18.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:39:18.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:39:18.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:18.003 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:39:18.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:39:18.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:18.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:18.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:18.863 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:18.859+0000 7f539762a7c0 -1 Falling back to public interface 2024-09-15T21:39:19.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:19.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:19.456 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:39:19.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:39:19.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:19.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:19.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:20.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:20.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:20.870 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:39:20.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:39:20.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:20.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:21.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:21.837 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:21.835+0000 7f539762a7c0 -1 osd.6 0 log_to_monitors true 2024-09-15T21:39:22.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:22.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:22.316 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:39:22.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:39:22.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:22.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:22.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:23.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:23.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:23.773 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:39:23.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:39:23.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:23.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:24.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:25.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:25.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:25.245 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:39:25.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:39:25.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:25.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:25.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:26.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:26.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:26.740 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:39:26.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:39:26.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:26.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:27.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:28.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:28.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:28.175 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:39:28.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:39:28.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:28.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:28.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:29.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:29.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:29.622 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:39:29.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:39:29.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:29.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:30.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:31.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:31.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:31.142 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:39:31.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:39:31.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:31.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:31.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:32.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:32.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:32.610 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:39:32.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:39:32.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:32.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:33.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:34.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:34.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:34.178 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:39:34.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:39:34.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:34.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:34.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:35.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:35.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:35.613 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:39:35.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:39:35.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:35.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:36.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:37.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:37.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:37.502 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:39:37.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:39:37.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:37.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:38.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:39.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:39.188 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:39:39.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:39.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:39:39.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:39.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:40.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:41.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:41.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:41.594 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:39:41.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:39:41.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:41.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:42.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:43.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:43.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:43.439 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:39:43.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:39:43.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:43.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:43.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:44.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:44.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:44.861 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:39:44.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:39:44.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:44.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:45.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:46.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:46.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:46.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:39:46.428 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:39:46.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:46.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:46.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:47.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:47.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:47.909 INFO:tasks.workunit.client.0.smithi104.stdout:19 2024-09-15T21:39:47.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:39:47.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:47.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:39:48.428 INFO:tasks.workunit.client.0.smithi104.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2471467738,v1:127.0.0.1:6851/2471467738] [v2:127.0.0.1:6852/2471467738,v1:127.0.0.1:6853/2471467738] exists,up 8f04fbd0-c08f-4b32-bb5e-b3d10aa3bc9b 2024-09-15T21:39:48.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:39:48.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:39:48.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:39:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T21:39:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2024-09-15T21:39:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-15T21:39:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:39:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-15T21:39:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:39:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/7 2024-09-15T21:39:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:39:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:39:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:39:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:39:48.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2024-09-15T21:39:48.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2024-09-15T21:39:48.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:39:48.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:39:48.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:39:48.433 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:39:48.433 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:39:48.433 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:39:48.433 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:39:48.433 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:39:48.434 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:39:48.434 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:39:48.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:39:48.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:39:48.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:39:48.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:39:48.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:39:48.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:39:48.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:39:48.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:39:48.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:39:48.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:39:48.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:39:48.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:39:48.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/7 2024-09-15T21:39:48.435 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:39:48.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3abbfbb1-3cbb-4fa8-bc6c-f933f1838bc3 2024-09-15T21:39:48.436 INFO:tasks.workunit.client.0.smithi104.stdout:add osd7 3abbfbb1-3cbb-4fa8-bc6c-f933f1838bc3 2024-09-15T21:39:48.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 3abbfbb1-3cbb-4fa8-bc6c-f933f1838bc3' 2024-09-15T21:39:48.436 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:39:48.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAkVOdmPbzjGhAA4UFAkzbLh/GjhtNOmP8+jQ== 2024-09-15T21:39:48.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAkVOdmPbzjGhAA4UFAkzbLh/GjhtNOmP8+jQ=="}' 2024-09-15T21:39:48.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3abbfbb1-3cbb-4fa8-bc6c-f933f1838bc3 -i td/test-erasure-code/7/new.json 2024-09-15T21:39:48.948 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:39:48.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/7/new.json 2024-09-15T21:39:48.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkVOdmPbzjGhAA4UFAkzbLh/GjhtNOmP8+jQ== --osd-uuid 3abbfbb1-3cbb-4fa8-bc6c-f933f1838bc3 2024-09-15T21:39:48.999 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:48.995+0000 7fe5384a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:49.012 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:49.007+0000 7fe5384a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:49.013 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:49.007+0000 7fe5384a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:49.013 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:49.007+0000 7fe5384a57c0 -1 bdev(0x55f57ed97c00 td/test-erasure-code/7/block) open stat got: (1) Operation not permitted 2024-09-15T21:39:49.013 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:49.007+0000 7fe5384a57c0 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2024-09-15T21:39:55.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/7/keyring 2024-09-15T21:39:55.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:39:55.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-15T21:39:55.452 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd7 key to auth repository 2024-09-15T21:39:55.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:39:56.052 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.7 2024-09-15T21:39:56.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-15T21:39:56.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:39:56.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:39:56.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:39:56.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:39:56.085 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:56.082+0000 7f4b2b5ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:56.120 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:56.118+0000 7f4b2b5ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:56.122 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:56.118+0000 7f4b2b5ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:56.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-15T21:39:56.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:39:56.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-15T21:39:56.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:39:56.499 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:39:56.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:39:56.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:56.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:39:56.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:56.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:39:56.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:57.655 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:39:57.650+0000 7f4b2b5ed7c0 -1 Falling back to public interface 2024-09-15T21:39:57.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:57.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:57.961 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:39:57.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:39:57.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:57.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:39:58.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:59.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:59.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:59.428 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:39:59.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:39:59.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:39:59.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:59.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:00.595 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:40:00.590+0000 7f4b2b5ed7c0 -1 osd.7 0 log_to_monitors true 2024-09-15T21:40:00.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:00.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:00.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:40:00.915 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:40:00.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:00.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:01.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:02.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:02.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:02.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:40:02.494 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:40:02.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:02.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:02.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:03.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:03.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:03.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:40:03.914 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:40:03.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:03.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:04.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:05.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:05.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:05.406 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:40:05.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:40:05.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:05.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:05.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:06.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:06.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:06.904 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:40:06.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:40:06.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:06.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:07.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:08.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:08.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:08.499 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:40:08.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:40:08.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:08.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:08.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:09.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:09.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:09.945 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:40:09.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:40:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:09.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:10.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:11.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:11.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:11.414 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:40:11.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:40:11.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:11.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:11.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:12.944 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:40:12.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:40:12.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:12.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:13.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:14.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:14.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:14.391 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:40:14.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:40:14.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:14.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:15.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:16.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:16.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:16.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:40:16.443 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:40:16.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:16.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:17.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:18.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:18.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:18.387 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:40:18.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:40:18.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:18.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:19.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:20.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:20.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:20.275 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:40:20.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:40:20.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:20.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:21.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:22.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:22.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:22.250 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:40:22.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:40:22.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:22.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:23.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:24.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:24.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:24.193 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:40:24.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:40:24.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:24.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:25.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:26.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:26.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:26.065 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:40:26.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:40:26.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:26.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:40:26.497 INFO:tasks.workunit.client.0.smithi104.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1419089862,v1:127.0.0.1:6859/1419089862] [v2:127.0.0.1:6860/1419089862,v1:127.0.0.1:6861/1419089862] exists,up 3abbfbb1-3cbb-4fa8-bc6c-f933f1838bc3 2024-09-15T21:40:26.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:40:26.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:40:26.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:40:26.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T21:40:26.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2024-09-15T21:40:26.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-15T21:40:26.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:40:26.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-15T21:40:26.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:40:26.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/8 2024-09-15T21:40:26.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:40:26.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:40:26.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:40:26.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:40:26.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2024-09-15T21:40:26.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2024-09-15T21:40:26.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:40:26.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:40:26.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:40:26.500 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:40:26.500 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:40:26.500 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:40:26.502 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:40:26.502 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:40:26.502 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:40:26.502 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:40:26.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:40:26.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:40:26.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:40:26.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:40:26.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:40:26.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:40:26.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:40:26.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:40:26.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:40:26.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:40:26.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:40:26.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:40:26.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/8 2024-09-15T21:40:26.504 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:40:26.504 INFO:tasks.workunit.client.0.smithi104.stdout:add osd8 52e62eec-3025-46af-a00e-3d37ccfe20d8 2024-09-15T21:40:26.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=52e62eec-3025-46af-a00e-3d37ccfe20d8 2024-09-15T21:40:26.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 52e62eec-3025-46af-a00e-3d37ccfe20d8' 2024-09-15T21:40:26.504 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:40:26.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBKVOdmqo/jHhAAjaatfhvMNln8pByKT3CySQ== 2024-09-15T21:40:26.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBKVOdmqo/jHhAAjaatfhvMNln8pByKT3CySQ=="}' 2024-09-15T21:40:26.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 52e62eec-3025-46af-a00e-3d37ccfe20d8 -i td/test-erasure-code/8/new.json 2024-09-15T21:40:27.136 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:40:27.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/8/new.json 2024-09-15T21:40:27.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKVOdmqo/jHhAAjaatfhvMNln8pByKT3CySQ== --osd-uuid 52e62eec-3025-46af-a00e-3d37ccfe20d8 2024-09-15T21:40:27.186 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:40:27.182+0000 7fc542aa87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:27.188 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:40:27.186+0000 7fc542aa87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:27.190 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:40:27.186+0000 7fc542aa87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:27.190 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:40:27.186+0000 7fc542aa87c0 -1 bdev(0x561f5f487c00 td/test-erasure-code/8/block) open stat got: (1) Operation not permitted 2024-09-15T21:40:27.190 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:40:27.186+0000 7fc542aa87c0 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2024-09-15T21:40:33.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/8/keyring 2024-09-15T21:40:33.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:40:33.494 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd8 key to auth repository 2024-09-15T21:40:33.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-15T21:40:33.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:40:34.062 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.8 2024-09-15T21:40:34.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-15T21:40:34.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:40:34.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:40:34.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:40:34.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:40:34.093 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:40:34.090+0000 7f181b41e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:34.129 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:40:34.126+0000 7f181b41e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:34.130 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:40:34.126+0000 7f181b41e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:34.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-15T21:40:34.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:40:34.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-15T21:40:34.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:40:34.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:40:34.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:34.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:40:34.524 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:40:34.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:34.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:35.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:40:35.902+0000 7f181b41e7c0 -1 Falling back to public interface 2024-09-15T21:40:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:36.011 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:40:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:40:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:36.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:37.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:37.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:37.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:40:37.423 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:40:37.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:37.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:37.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:38.616 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:40:38.614+0000 7f181b41e7c0 -1 osd.8 0 log_to_monitors true 2024-09-15T21:40:38.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:38.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:38.859 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:40:38.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:40:38.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:38.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:39.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:40.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:40.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:40.552 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:40:40.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:40:40.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:40.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:40.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:41.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:41.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:41.996 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:40:41.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:40:41.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:41.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:42.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:43.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:43.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:43.416 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:40:43.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:40:43.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:43.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:43.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:44.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:44.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:44.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:40:44.896 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:40:44.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:44.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:45.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:46.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:46.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:46.378 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:40:46.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:40:46.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:46.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:46.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:47.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:47.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:47.911 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:40:47.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:40:47.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:47.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:48.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:49.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:49.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:49.335 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:40:49.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:40:49.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:49.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:49.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:50.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:50.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:50.908 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:40:50.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:40:50.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:50.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:51.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:52.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:52.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:52.375 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:40:52.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:40:52.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:52.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:53.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:54.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:54.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:54.374 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:40:54.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:40:54.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:54.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:55.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:56.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:56.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:56.385 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:40:56.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:40:56.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:56.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:56.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:57.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:57.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:57.986 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:40:57.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:40:57.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:57.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:40:58.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:59.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:59.511 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:40:59.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:59.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:40:59.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:59.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:41:00.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:01.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:01.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:41:01.871 INFO:tasks.workunit.client.0.smithi104.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/985200777,v1:127.0.0.1:6867/985200777] [v2:127.0.0.1:6868/985200777,v1:127.0.0.1:6869/985200777] exists,up 52e62eec-3025-46af-a00e-3d37ccfe20d8 2024-09-15T21:41:01.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:41:01.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:41:01.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/9 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-15T21:41:01.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-15T21:41:01.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:41:01.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:41:01.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:41:01.876 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:41:01.876 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:41:01.876 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:41:01.876 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:41:01.876 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:01.876 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:41:01.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:41:01.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/9 2024-09-15T21:41:01.896 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:41:01.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=86ff2c29-70ab-4d48-bd99-44b68dc774a0 2024-09-15T21:41:01.897 INFO:tasks.workunit.client.0.smithi104.stdout:add osd9 86ff2c29-70ab-4d48-bd99-44b68dc774a0 2024-09-15T21:41:01.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 86ff2c29-70ab-4d48-bd99-44b68dc774a0' 2024-09-15T21:41:01.898 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:41:01.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBtVOdmSbN6NhAARtecbKAfu/L8SOzo9P2xUQ== 2024-09-15T21:41:01.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBtVOdmSbN6NhAARtecbKAfu/L8SOzo9P2xUQ=="}' 2024-09-15T21:41:01.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 86ff2c29-70ab-4d48-bd99-44b68dc774a0 -i td/test-erasure-code/9/new.json 2024-09-15T21:41:02.574 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:41:02.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/9/new.json 2024-09-15T21:41:02.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtVOdmSbN6NhAARtecbKAfu/L8SOzo9P2xUQ== --osd-uuid 86ff2c29-70ab-4d48-bd99-44b68dc774a0 2024-09-15T21:41:02.625 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:02.622+0000 7f80e73197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:02.660 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:02.658+0000 7f80e73197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:02.661 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:02.658+0000 7f80e73197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:02.661 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:02.658+0000 7f80e73197c0 -1 bdev(0x55f38978fc00 td/test-erasure-code/9/block) open stat got: (1) Operation not permitted 2024-09-15T21:41:02.662 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:02.658+0000 7f80e73197c0 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2024-09-15T21:41:08.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/9/keyring 2024-09-15T21:41:08.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:41:08.574 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd9 key to auth repository 2024-09-15T21:41:08.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-15T21:41:08.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:41:09.179 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.9 2024-09-15T21:41:09.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-15T21:41:09.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:41:09.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:41:09.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:41:09.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:41:09.211 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:09.206+0000 7fd932b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:09.237 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:09.234+0000 7fd932b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:09.238 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:09.234+0000 7fd932b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:09.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-15T21:41:09.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:41:09.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:41:09.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:41:09.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:41:09.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:09.642 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:41:09.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:41:09.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:09.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:10.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:10.290 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:10.286+0000 7fd932b497c0 -1 Falling back to public interface 2024-09-15T21:41:11.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:11.136 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:41:11.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:11.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:41:11.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:11.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:11.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:12.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:12.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:12.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:41:12.565 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:41:12.566 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:12.566 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:12.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:13.358 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:13.354+0000 7fd932b497c0 -1 osd.9 0 log_to_monitors true 2024-09-15T21:41:13.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:13.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:13.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:41:13.985 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:41:13.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:13.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:14.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:15.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:15.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:15.552 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:41:15.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:41:15.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:15.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:16.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:17.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:17.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:17.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:41:17.047 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:41:17.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:17.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:17.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:18.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:18.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:18.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:41:18.678 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:41:18.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:18.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:19.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:20.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:20.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:20.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:41:20.213 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:41:20.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:20.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:20.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:21.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:21.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:21.693 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:41:21.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:41:21.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:21.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:22.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:23.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:23.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:23.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:41:23.213 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:41:23.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:23.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:23.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:24.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:24.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:24.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:41:24.675 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:41:24.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:24.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:25.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:26.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:26.250 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:41:26.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:26.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:41:26.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:26.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:27.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:28.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:28.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:28.127 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:41:28.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:41:28.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:28.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:29.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:30.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:30.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:30.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:41:30.229 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:41:30.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:30.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:30.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:31.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:31.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:31.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:41:31.728 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:41:31.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:31.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:32.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:33.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:33.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:33.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:41:33.326 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:41:33.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:33.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:33.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:34.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:34.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:34.796 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:41:34.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:41:34.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:34.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:35.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:36.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:36.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:36.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:41:36.697 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:41:36.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:36.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:41:37.174 INFO:tasks.workunit.client.0.smithi104.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2116259112,v1:127.0.0.1:6875/2116259112] [v2:127.0.0.1:6876/2116259112,v1:127.0.0.1:6877/2116259112] exists,up 86ff2c29-70ab-4d48-bd99-44b68dc774a0 2024-09-15T21:41:37.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:41:37.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:41:37.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:41:37.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T21:41:37.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2024-09-15T21:41:37.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-15T21:41:37.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:41:37.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=10 2024-09-15T21:41:37.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:41:37.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/10 2024-09-15T21:41:37.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:41:37.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:41:37.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:41:37.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:41:37.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2024-09-15T21:41:37.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2024-09-15T21:41:37.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:41:37.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:41:37.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:41:37.185 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:41:37.185 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:41:37.185 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:41:37.185 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:41:37.185 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:37.186 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:41:37.186 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:41:37.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:41:37.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:41:37.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:41:37.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:41:37.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:41:37.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:41:37.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:41:37.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:41:37.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:41:37.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:41:37.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:41:37.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:41:37.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/10 2024-09-15T21:41:37.189 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:41:37.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4f22bbfc-0d8d-497a-935f-5c08d6f06ee1 2024-09-15T21:41:37.190 INFO:tasks.workunit.client.0.smithi104.stdout:add osd10 4f22bbfc-0d8d-497a-935f-5c08d6f06ee1 2024-09-15T21:41:37.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd10 4f22bbfc-0d8d-497a-935f-5c08d6f06ee1' 2024-09-15T21:41:37.191 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:41:37.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCRVOdmbIPiCxAAD9iu7CdI6UExoQTTo924yA== 2024-09-15T21:41:37.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCRVOdmbIPiCxAAD9iu7CdI6UExoQTTo924yA=="}' 2024-09-15T21:41:37.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4f22bbfc-0d8d-497a-935f-5c08d6f06ee1 -i td/test-erasure-code/10/new.json 2024-09-15T21:41:37.807 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:41:37.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/10/new.json 2024-09-15T21:41:37.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 10 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRVOdmbIPiCxAAD9iu7CdI6UExoQTTo924yA== --osd-uuid 4f22bbfc-0d8d-497a-935f-5c08d6f06ee1 2024-09-15T21:41:37.850 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:37.845+0000 7fc2d91aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:37.853 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:37.849+0000 7fc2d91aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:37.854 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:37.849+0000 7fc2d91aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:37.854 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:37.849+0000 7fc2d91aa7c0 -1 bdev(0x55c55ac2dc00 td/test-erasure-code/10/block) open stat got: (1) Operation not permitted 2024-09-15T21:41:37.854 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:37.849+0000 7fc2d91aa7c0 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2024-09-15T21:41:44.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/10/keyring 2024-09-15T21:41:44.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:41:44.318 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd10 key to auth repository 2024-09-15T21:41:44.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd10 key to auth repository 2024-09-15T21:41:44.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:41:44.901 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.10 2024-09-15T21:41:44.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.10 2024-09-15T21:41:44.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 10 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:41:44.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:41:44.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:41:44.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:41:44.932 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:44.929+0000 7fe0a263b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:44.934 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:44.929+0000 7fe0a263b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:44.935 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:44.929+0000 7fe0a263b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:45.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 10 2024-09-15T21:41:45.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:41:45.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=10 2024-09-15T21:41:45.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:41:45.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:41:45.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:45.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:41:45.317 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:41:45.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:45.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:41:45.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:46.027 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:46.021+0000 7fe0a263b7c0 -1 Falling back to public interface 2024-09-15T21:41:46.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:46.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:46.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:41:46.770 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:41:46.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:46.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:41:47.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:48.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:48.198 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:41:48.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:48.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:41:48.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:48.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:41:48.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:48.762 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:41:48.757+0000 7fe0a263b7c0 -1 osd.10 0 log_to_monitors true 2024-09-15T21:41:49.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:49.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:49.631 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:41:49.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:41:49.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:49.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:41:50.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:51.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:51.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:51.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:41:51.305 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:41:51.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:51.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:41:51.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:52.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:52.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:52.761 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:41:52.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:41:52.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:52.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:41:53.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:54.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:54.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:54.221 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:41:54.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:41:54.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:54.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:41:54.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:55.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:55.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:55.724 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:41:55.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:41:55.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:55.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:41:56.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:57.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:57.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:57.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:41:57.169 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:41:57.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:57.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:41:57.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:58.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:58.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:58.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:41:58.694 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:41:58.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:58.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:41:59.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:00.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:00.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:00.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:42:00.125 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:42:00.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:00.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:42:00.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:01.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:01.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:01.916 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:42:01.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:42:01.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:01.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:42:03.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:04.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:04.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:04.081 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:42:04.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:42:04.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:04.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:42:05.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:06.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:06.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:06.147 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:42:06.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:42:06.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:06.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:42:07.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:08.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:08.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:08.266 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:42:08.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:42:08.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:08.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:42:09.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:10.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:10.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:10.169 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:42:10.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:42:10.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:10.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:42:11.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:12.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:12.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:12.055 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:42:12.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:42:12.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:12.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-15T21:42:12.557 INFO:tasks.workunit.client.0.smithi104.stdout:osd.10 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/3313472993,v1:127.0.0.1:6883/3313472993] [v2:127.0.0.1:6884/3313472993,v1:127.0.0.1:6885/3313472993] exists,up 4f22bbfc-0d8d-497a-935f-5c08d6f06ee1 2024-09-15T21:42:12.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:42:12.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:42:12.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:42:12.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2024-09-15T21:42:12.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:42:13.193 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' does not exist 2024-09-15T21:42:13.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:42:13.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:42:13.886 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T21:42:13.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:42:14.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:42:18.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2024-09-15T21:42:18.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:42:18.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:42:18.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:42:18.961 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:42:18.962 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:42:18.962 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:42:18.962 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:42:18.962 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:42:18.962 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:42:19.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:42:19.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:42:19.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:42:19.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:42:19.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:42:19.079 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:42:19.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:8 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:42:19.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:19.527 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:42:19.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836561 2024-09-15T21:42:19.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836561 2024-09-15T21:42:19.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561' 2024-09-15T21:42:19.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:19.685 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:42:19.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673034 2024-09-15T21:42:19.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673034 2024-09-15T21:42:19.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673034' 2024-09-15T21:42:19.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:19.842 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:42:19.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509509 2024-09-15T21:42:19.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509509 2024-09-15T21:42:19.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673034 2-64424509509' 2024-09-15T21:42:19.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:19.996 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:42:20.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345983 2024-09-15T21:42:20.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345983 2024-09-15T21:42:20.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673034 2-64424509509 3-85899345983' 2024-09-15T21:42:20.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:20.147 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:42:20.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182451 2024-09-15T21:42:20.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182451 2024-09-15T21:42:20.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673034 2-64424509509 3-85899345983 4-107374182451' 2024-09-15T21:42:20.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:20.296 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:42:20.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018930 2024-09-15T21:42:20.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018930 2024-09-15T21:42:20.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673034 2-64424509509 3-85899345983 4-107374182451 5-128849018930' 2024-09-15T21:42:20.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:20.443 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:42:20.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855401 2024-09-15T21:42:20.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855401 2024-09-15T21:42:20.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673034 2-64424509509 3-85899345983 4-107374182451 5-128849018930 6-150323855401' 2024-09-15T21:42:20.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:20.594 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:42:20.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691868 2024-09-15T21:42:20.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691868 2024-09-15T21:42:20.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673034 2-64424509509 3-85899345983 4-107374182451 5-128849018930 6-150323855401 7-171798691868' 2024-09-15T21:42:20.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:20.746 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:42:20.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528345 2024-09-15T21:42:20.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528345 2024-09-15T21:42:20.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673034 2-64424509509 3-85899345983 4-107374182451 5-128849018930 6-150323855401 7-171798691868 8-193273528345' 2024-09-15T21:42:20.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:20.895 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:42:21.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364816 2024-09-15T21:42:21.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364816 2024-09-15T21:42:21.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673034 2-64424509509 3-85899345983 4-107374182451 5-128849018930 6-150323855401 7-171798691868 8-193273528345 9-214748364816' 2024-09-15T21:42:21.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:21.046 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T21:42:21.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201291 2024-09-15T21:42:21.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201291 2024-09-15T21:42:21.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673034 2-64424509509 3-85899345983 4-107374182451 5-128849018930 6-150323855401 7-171798691868 8-193273528345 9-214748364816 10-236223201291' 2024-09-15T21:42:21.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:21.196 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836561 2024-09-15T21:42:21.196 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:21.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:42:21.197 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836561 2024-09-15T21:42:21.197 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:21.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836561 2024-09-15T21:42:21.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-09-15T21:42:21.198 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 21474836561 2024-09-15T21:42:21.199 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:42:21.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836562 -lt 21474836561 2024-09-15T21:42:21.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:21.644 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673034 2024-09-15T21:42:21.644 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:21.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:42:21.647 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673034 2024-09-15T21:42:21.647 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:21.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673034 2024-09-15T21:42:21.648 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 42949673034 2024-09-15T21:42:21.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673034' 2024-09-15T21:42:21.649 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:42:22.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673034 -lt 42949673034 2024-09-15T21:42:22.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:22.092 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509509 2024-09-15T21:42:22.093 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:22.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:42:22.095 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509509 2024-09-15T21:42:22.095 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:22.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509509 2024-09-15T21:42:22.097 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 64424509509 2024-09-15T21:42:22.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509509' 2024-09-15T21:42:22.098 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:42:22.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509509 -lt 64424509509 2024-09-15T21:42:22.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:22.620 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345983 2024-09-15T21:42:22.621 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:22.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:42:22.623 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345983 2024-09-15T21:42:22.623 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:22.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345983 2024-09-15T21:42:22.624 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 85899345983 2024-09-15T21:42:22.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345983' 2024-09-15T21:42:22.625 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:42:23.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345983 -lt 85899345983 2024-09-15T21:42:23.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:23.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182451 2024-09-15T21:42:23.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:23.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:42:23.273 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182451 2024-09-15T21:42:23.273 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:23.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182451 2024-09-15T21:42:23.278 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 107374182451 2024-09-15T21:42:23.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-09-15T21:42:23.278 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:42:23.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182452 -lt 107374182451 2024-09-15T21:42:23.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:23.724 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018930 2024-09-15T21:42:23.724 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:23.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:42:23.726 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018930 2024-09-15T21:42:23.727 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:23.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018930 2024-09-15T21:42:23.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018930' 2024-09-15T21:42:23.728 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 128849018930 2024-09-15T21:42:23.729 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:42:24.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018930 -lt 128849018930 2024-09-15T21:42:24.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:24.225 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:24.225 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855401 2024-09-15T21:42:24.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:42:24.226 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855401 2024-09-15T21:42:24.227 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:24.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855401 2024-09-15T21:42:24.228 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 150323855401 2024-09-15T21:42:24.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2024-09-15T21:42:24.228 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:42:24.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855402 -lt 150323855401 2024-09-15T21:42:24.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:24.697 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691868 2024-09-15T21:42:24.697 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:24.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:42:24.699 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691868 2024-09-15T21:42:24.699 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:24.700 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.7 seq 171798691868 2024-09-15T21:42:24.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691868 2024-09-15T21:42:24.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-09-15T21:42:24.701 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:42:25.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691869 -lt 171798691868 2024-09-15T21:42:25.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:25.156 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:25.157 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528345 2024-09-15T21:42:25.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:42:25.159 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528345 2024-09-15T21:42:25.159 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:25.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528345 2024-09-15T21:42:25.159 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.8 seq 193273528345 2024-09-15T21:42:25.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-09-15T21:42:25.160 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:42:25.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528346 -lt 193273528345 2024-09-15T21:42:25.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:25.594 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364816 2024-09-15T21:42:25.595 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:25.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:42:25.597 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364816 2024-09-15T21:42:25.598 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:25.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364816 2024-09-15T21:42:25.599 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.9 seq 214748364816 2024-09-15T21:42:25.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364816' 2024-09-15T21:42:25.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:42:26.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364817 -lt 214748364816 2024-09-15T21:42:26.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:26.141 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201291 2024-09-15T21:42:26.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:26.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-15T21:42:26.145 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201291 2024-09-15T21:42:26.145 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:26.146 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.10 seq 236223201291 2024-09-15T21:42:26.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201291 2024-09-15T21:42:26.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201291' 2024-09-15T21:42:26.147 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T21:42:26.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201292 -lt 236223201291 2024-09-15T21:42:26.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:42:26.635 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:42:26.635 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:42:27.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:42:27.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:42:27.300 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:42:27.300 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:42:27.300 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:42:27.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:42:27.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:42:27.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:42:27.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:42:27.708 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:42:27.709 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:42:27.709 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:42:28.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:42:28.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:42:28.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:42:28.272 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2024-09-15T21:42:28.272 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:42:28.272 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:42:28.273 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:42:28.273 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:42:28.273 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:42:28.273 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.50061/ceph-osd.0.asok 2024-09-15T21:42:28.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2024-09-15T21:42:28.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.50061/ceph-osd.0.asok log flush 2024-09-15T21:42:28.372 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:42:28.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2024-09-15T21:42:28.390 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:35:32.645+0000 7f8de48ec7c0 0 load: jerasure load: lrc 2024-09-15T21:42:28.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2024-09-15T21:42:28.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2024-09-15T21:42:28.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2024-09-15T21:42:28.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2024-09-15T21:42:28.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2024-09-15T21:42:29.900 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'ecpool' already exists 2024-09-15T21:42:29.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:42:30.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2024-09-15T21:42:30.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:42:30.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:42:30.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:42:30.921 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:42:30.922 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:42:30.922 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:42:30.923 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:42:30.923 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:42:30.923 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:42:31.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:42:31.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:42:31.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:42:31.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:42:31.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:42:31.041 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:42:32.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:42:32.147 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:42:32.148 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:42:32.148 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:42:32.148 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:42:32.148 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:42:32.148 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:42:32.148 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:42:32.148 INFO:tasks.workunit.client.0.smithi104.stderr:8 2024-09-15T21:42:32.148 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:42:32.148 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:42:32.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:42:32.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:32.149 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:42:32.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836566 2024-09-15T21:42:32.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836566 2024-09-15T21:42:32.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566' 2024-09-15T21:42:32.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:32.809 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:42:32.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673038 2024-09-15T21:42:32.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673038 2024-09-15T21:42:32.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673038' 2024-09-15T21:42:32.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:32.968 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:42:33.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509514 2024-09-15T21:42:33.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509514 2024-09-15T21:42:33.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509514' 2024-09-15T21:42:33.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:33.273 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:42:33.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345987 2024-09-15T21:42:33.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345987 2024-09-15T21:42:33.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509514 3-85899345987' 2024-09-15T21:42:33.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:33.589 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:42:33.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182456 2024-09-15T21:42:33.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182456 2024-09-15T21:42:33.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509514 3-85899345987 4-107374182456' 2024-09-15T21:42:33.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:33.861 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:42:34.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018934 2024-09-15T21:42:34.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018934 2024-09-15T21:42:34.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509514 3-85899345987 4-107374182456 5-128849018934' 2024-09-15T21:42:34.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:34.117 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:42:34.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855406 2024-09-15T21:42:34.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855406 2024-09-15T21:42:34.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509514 3-85899345987 4-107374182456 5-128849018934 6-150323855406' 2024-09-15T21:42:34.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:34.281 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:42:34.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691873 2024-09-15T21:42:34.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691873 2024-09-15T21:42:34.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509514 3-85899345987 4-107374182456 5-128849018934 6-150323855406 7-171798691873' 2024-09-15T21:42:34.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:34.500 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:42:34.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528350 2024-09-15T21:42:34.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528350 2024-09-15T21:42:34.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509514 3-85899345987 4-107374182456 5-128849018934 6-150323855406 7-171798691873 8-193273528350' 2024-09-15T21:42:34.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:34.662 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:42:34.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364821 2024-09-15T21:42:34.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364821 2024-09-15T21:42:34.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509514 3-85899345987 4-107374182456 5-128849018934 6-150323855406 7-171798691873 8-193273528350 9-214748364821' 2024-09-15T21:42:34.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:34.815 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T21:42:34.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201296 2024-09-15T21:42:34.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201296 2024-09-15T21:42:34.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673038 2-64424509514 3-85899345987 4-107374182456 5-128849018934 6-150323855406 7-171798691873 8-193273528350 9-214748364821 10-236223201296' 2024-09-15T21:42:34.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:34.968 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836566 2024-09-15T21:42:34.968 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:34.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:42:34.969 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836566 2024-09-15T21:42:34.970 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:34.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836566 2024-09-15T21:42:34.970 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 21474836566 2024-09-15T21:42:34.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836566' 2024-09-15T21:42:34.971 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:42:35.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836566 -lt 21474836566 2024-09-15T21:42:35.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:35.423 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673038 2024-09-15T21:42:35.423 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:35.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:42:35.425 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673038 2024-09-15T21:42:35.425 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:35.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673038 2024-09-15T21:42:35.426 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 42949673038 2024-09-15T21:42:35.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-09-15T21:42:35.427 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:42:35.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673039 -lt 42949673038 2024-09-15T21:42:35.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:35.896 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509514 2024-09-15T21:42:35.896 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:35.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:42:35.898 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:35.898 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509514 2024-09-15T21:42:35.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509514 2024-09-15T21:42:35.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-09-15T21:42:35.900 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 64424509514 2024-09-15T21:42:35.900 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:42:36.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509514 -lt 64424509514 2024-09-15T21:42:36.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:36.436 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345987 2024-09-15T21:42:36.437 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:36.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:42:36.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345987 2024-09-15T21:42:36.439 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:36.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345987 2024-09-15T21:42:36.440 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 85899345987 2024-09-15T21:42:36.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345987' 2024-09-15T21:42:36.441 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:42:36.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345988 -lt 85899345987 2024-09-15T21:42:36.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:36.885 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182456 2024-09-15T21:42:36.885 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:36.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:42:36.886 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182456 2024-09-15T21:42:36.886 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:36.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182456 2024-09-15T21:42:36.887 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 107374182456 2024-09-15T21:42:36.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182456' 2024-09-15T21:42:36.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:42:37.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182457 -lt 107374182456 2024-09-15T21:42:37.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:37.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:37.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018934 2024-09-15T21:42:37.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:42:37.330 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018934 2024-09-15T21:42:37.330 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:37.331 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018934 2024-09-15T21:42:37.331 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 128849018934 2024-09-15T21:42:37.331 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018934' 2024-09-15T21:42:37.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:42:37.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018935 -lt 128849018934 2024-09-15T21:42:37.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:37.785 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855406 2024-09-15T21:42:37.785 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:37.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:42:37.788 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855406 2024-09-15T21:42:37.788 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:37.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855406 2024-09-15T21:42:37.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855406' 2024-09-15T21:42:37.789 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 150323855406 2024-09-15T21:42:37.790 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:42:38.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855406 -lt 150323855406 2024-09-15T21:42:38.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:38.222 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691873 2024-09-15T21:42:38.223 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:38.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:42:38.225 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:38.225 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691873 2024-09-15T21:42:38.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691873 2024-09-15T21:42:38.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-09-15T21:42:38.226 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.7 seq 171798691873 2024-09-15T21:42:38.226 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:42:38.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691874 -lt 171798691873 2024-09-15T21:42:38.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:38.696 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528350 2024-09-15T21:42:38.697 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:38.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:42:38.698 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528350 2024-09-15T21:42:38.699 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:38.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528350 2024-09-15T21:42:38.700 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.8 seq 193273528350 2024-09-15T21:42:38.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-09-15T21:42:38.701 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:42:39.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528350 -lt 193273528350 2024-09-15T21:42:39.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:39.148 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364821 2024-09-15T21:42:39.148 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:39.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:42:39.150 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364821 2024-09-15T21:42:39.150 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:39.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364821 2024-09-15T21:42:39.151 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.9 seq 214748364821 2024-09-15T21:42:39.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364821' 2024-09-15T21:42:39.152 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:42:39.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364822 -lt 214748364821 2024-09-15T21:42:39.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:39.565 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:39.565 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201296 2024-09-15T21:42:39.566 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-15T21:42:39.567 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:39.567 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201296 2024-09-15T21:42:39.568 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.10 seq 236223201296 2024-09-15T21:42:39.568 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201296 2024-09-15T21:42:39.568 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201296' 2024-09-15T21:42:39.569 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T21:42:40.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201297 -lt 236223201296 2024-09-15T21:42:40.026 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:42:40.026 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:42:40.027 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:42:40.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-15T21:42:40.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:42:40.594 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:42:40.594 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:42:40.594 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:42:40.594 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:42:40.594 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:42:40.594 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:42:41.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-15T21:42:41.019 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:42:41.019 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:42:41.019 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:42:41.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-15T21:42:41.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:42:41.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:42:41.630 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2024-09-15T21:42:41.631 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T21:42:41.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2024-09-15T21:42:41.633 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_chunk_mapping 2024-09-15T21:42:41.633 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_put_get_isa 2024-09-15T21:42:41.633 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_put_get_jerasure 2024-09-15T21:42:41.633 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_put_get_lrc_advanced 2024-09-15T21:42:41.634 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_put_get_lrc_kml 2024-09-15T21:42:41.634 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_put_get_shec' 2024-09-15T21:42:41.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T21:42:41.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2024-09-15T21:42:41.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: TEST_alignment_constraints: local payload=ABC 2024-09-15T21:42:41.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: TEST_alignment_constraints: echo ABC 2024-09-15T21:42:41.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-15T21:42:41.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: local stripe_unit=4096 2024-09-15T21:42:41.648 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2024-09-15T21:42:41.649 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: grep k= 2024-09-15T21:42:42.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: eval local k=2 2024-09-15T21:42:42.133 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: local k=2 2024-09-15T21:42:42.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: local block_size=8191 2024-09-15T21:42:42.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:263: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2024-09-15T21:42:42.134 INFO:tasks.workunit.client.0.smithi104.stderr:2+0 records in 2024-09-15T21:42:42.134 INFO:tasks.workunit.client.0.smithi104.stderr:2+0 records out 2024-09-15T21:42:42.134 INFO:tasks.workunit.client.0.smithi104.stderr:16382 bytes (16 kB, 16 KiB) copied, 0.000125017 s, 131 MB/s 2024-09-15T21:42:42.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:264: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2024-09-15T21:42:42.165 INFO:tasks.workunit.client.0.smithi104.stderr:INFO: op_size has been rounded to 8192 2024-09-15T21:42:42.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2024-09-15T21:42:42.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T21:42:42.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2024-09-15T21:42:42.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: TEST_chunk_mapping: local dir=td/test-erasure-code 2024-09-15T21:42:42.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:311: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2024-09-15T21:42:42.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-15T21:42:42.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=ecpool 2024-09-15T21:42:42.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=0 2024-09-15T21:42:42.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=1 2024-09-15T21:42:42.563 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-15T21:42:42.564 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-15T21:42:42.584 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-15T21:42:42.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2024-09-15T21:42:42.585 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-15T21:42:42.585 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-15T21:42:42.609 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-15T21:42:42.609 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2024-09-15T21:42:42.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2024-09-15T21:42:42.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2024-09-15T21:42:42.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2024-09-15T21:42:42.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2024-09-15T21:42:42.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-15T21:42:42.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:42:42.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGecpool 2024-09-15T21:42:42.935 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:42:42.935 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-15T21:42:43.372 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T21:42:43.373 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:42:43.373 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:42:43.373 INFO:tasks.workunit.client.0.smithi104.stderr:0' 2024-09-15T21:42:43.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 2024-09-15T21:42:43.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-09-15T21:42:43.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-09-15T21:42:43.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-09-15T21:42:43.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-15T21:42:43.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2024-09-15T21:42:43.498 INFO:tasks.workunit.client.0.smithi104.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T21:42:43.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-15T21:42:43.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-15T21:42:43.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2024-09-15T21:42:43.627 INFO:tasks.workunit.client.0.smithi104.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T21:42:43.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-15T21:42:43.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-15T21:42:43.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-15T21:42:43.760 INFO:tasks.workunit.client.0.smithi104.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T21:42:43.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-15T21:42:43.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-15T21:42:43.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2024-09-15T21:42:43.892 INFO:tasks.workunit.client.0.smithi104.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T21:42:43.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-15T21:42:43.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-15T21:42:43.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:42:43.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-15T21:42:43.905 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-15T21:42:43.905 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:42:43.905 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGecpool 2024-09-15T21:42:43.905 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-15T21:42:43.905 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:42:44.382 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T21:42:44.382 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:42:44.382 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:42:44.382 INFO:tasks.workunit.client.0.smithi104.stderr:0' 2024-09-15T21:42:44.382 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 2024-09-15T21:42:44.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-09-15T21:42:44.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-09-15T21:42:44.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-15T21:42:44.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTecpool 2024-09-15T21:42:44.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-15T21:42:44.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:42:44.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T21:42:44.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:42:44.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-15T21:42:44.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-15T21:42:44.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:42:44.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T21:42:44.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:42:44.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2024-09-15T21:42:44.385 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:42:44.385 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:42:44.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:42:44.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:42:44.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:42:44.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:42:44.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-15T21:42:44.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-15T21:42:44.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:44.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T21:42:44.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:44.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2024-09-15T21:42:44.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2024-09-15T21:42:45.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2024-09-15T21:42:45.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/3 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:42:45.946 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:42:45.947 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:42:45.947 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:42:45.947 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:42:45.947 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:42:45.947 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:42:45.947 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:42:45.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:42:45.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:42:45.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/3 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.3 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:42:45.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/3/whoami 2024-09-15T21:42:45.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T21:42:45.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:42:45.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:42:45.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:42:45.984 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:42:45.981+0000 7f86a8f2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:42:45.987 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:42:45.981+0000 7f86a8f2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:42:45.989 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:42:45.985+0000 7f86a8f2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:42:46.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T21:42:46.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:42:46.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:42:46.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:42:46.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:42:46.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:46.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:42:46.430 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:42:46.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:46.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:42:46.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:47.230 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:42:47.225+0000 7f86a8f2b7c0 -1 Falling back to public interface 2024-09-15T21:42:47.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:47.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:47.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:42:47.876 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:42:47.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:47.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:42:48.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:49.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:49.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:49.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:42:49.351 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:42:49.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:49.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:42:49.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:50.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:50.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:50.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:42:50.816 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:42:50.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:50.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:42:50.853 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:42:50.849+0000 7f86a8f2b7c0 -1 osd.3 77 log_to_monitors true 2024-09-15T21:42:51.406 INFO:tasks.workunit.client.0.smithi104.stderr:osd.3 up in weight 1 up_from 81 up_thru 70 down_at 78 last_clean_interval [20,77) [v2:127.0.0.1:6826/3141427844,v1:127.0.0.1:6827/3141427844] [v2:127.0.0.1:6828/3141427844,v1:127.0.0.1:6829/3141427844] exists,up 7e795b35-bf62-4700-b20b-41066d193de7 2024-09-15T21:42:51.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:42:51.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:42:51.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:42:51.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:42:51.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:42:51.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:42:51.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:42:51.408 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:42:51.408 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:42:51.408 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:42:51.408 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:42:51.408 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:42:51.408 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:42:51.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:42:51.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:42:51.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:42:51.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:42:51.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:42:51.514 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:42:52.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:42:52.298 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:42:52.298 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:42:52.299 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:42:52.299 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:42:52.299 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:42:52.299 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:42:52.299 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:42:52.299 INFO:tasks.workunit.client.0.smithi104.stderr:8 2024-09-15T21:42:52.299 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:42:52.299 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:42:52.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:42:52.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:52.299 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:42:52.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836572 2024-09-15T21:42:52.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836572 2024-09-15T21:42:52.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572' 2024-09-15T21:42:52.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:52.601 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:42:52.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673044 2024-09-15T21:42:52.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673044 2024-09-15T21:42:52.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673044' 2024-09-15T21:42:52.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:52.802 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:42:52.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509520 2024-09-15T21:42:52.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509520 2024-09-15T21:42:52.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673044 2-64424509520' 2024-09-15T21:42:52.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:52.956 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:42:53.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350979 2024-09-15T21:42:53.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350979 2024-09-15T21:42:53.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673044 2-64424509520 3-347892350979' 2024-09-15T21:42:53.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:53.217 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:42:53.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182462 2024-09-15T21:42:53.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182462 2024-09-15T21:42:53.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673044 2-64424509520 3-347892350979 4-107374182462' 2024-09-15T21:42:53.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:53.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:42:53.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018940 2024-09-15T21:42:53.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018940 2024-09-15T21:42:53.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673044 2-64424509520 3-347892350979 4-107374182462 5-128849018940' 2024-09-15T21:42:53.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:53.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:42:53.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855412 2024-09-15T21:42:53.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855412 2024-09-15T21:42:53.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673044 2-64424509520 3-347892350979 4-107374182462 5-128849018940 6-150323855412' 2024-09-15T21:42:53.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:53.757 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:42:53.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691879 2024-09-15T21:42:53.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691879 2024-09-15T21:42:53.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673044 2-64424509520 3-347892350979 4-107374182462 5-128849018940 6-150323855412 7-171798691879' 2024-09-15T21:42:53.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:53.912 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:42:54.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528356 2024-09-15T21:42:54.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528356 2024-09-15T21:42:54.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673044 2-64424509520 3-347892350979 4-107374182462 5-128849018940 6-150323855412 7-171798691879 8-193273528356' 2024-09-15T21:42:54.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:54.072 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:42:54.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364827 2024-09-15T21:42:54.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364827 2024-09-15T21:42:54.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673044 2-64424509520 3-347892350979 4-107374182462 5-128849018940 6-150323855412 7-171798691879 8-193273528356 9-214748364827' 2024-09-15T21:42:54.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:54.224 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T21:42:54.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201302 2024-09-15T21:42:54.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201302 2024-09-15T21:42:54.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673044 2-64424509520 3-347892350979 4-107374182462 5-128849018940 6-150323855412 7-171798691879 8-193273528356 9-214748364827 10-236223201302' 2024-09-15T21:42:54.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:54.378 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836572 2024-09-15T21:42:54.378 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:54.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:42:54.380 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836572 2024-09-15T21:42:54.380 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:54.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836572 2024-09-15T21:42:54.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-09-15T21:42:54.381 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 21474836572 2024-09-15T21:42:54.382 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:42:54.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836572 -lt 21474836572 2024-09-15T21:42:54.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:54.800 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673044 2024-09-15T21:42:54.800 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:54.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:42:54.801 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673044 2024-09-15T21:42:54.801 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:54.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673044 2024-09-15T21:42:54.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-09-15T21:42:54.803 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 42949673044 2024-09-15T21:42:54.803 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:42:55.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673045 -lt 42949673044 2024-09-15T21:42:55.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:55.234 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509520 2024-09-15T21:42:55.234 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:55.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:42:55.235 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509520 2024-09-15T21:42:55.235 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:55.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509520 2024-09-15T21:42:55.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-09-15T21:42:55.237 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 64424509520 2024-09-15T21:42:55.237 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:42:55.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509520 -lt 64424509520 2024-09-15T21:42:55.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:55.665 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892350979 2024-09-15T21:42:55.666 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:55.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:42:55.668 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892350979 2024-09-15T21:42:55.668 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:55.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350979 2024-09-15T21:42:55.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892350979' 2024-09-15T21:42:55.670 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 347892350979 2024-09-15T21:42:55.670 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:42:56.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350979 -lt 347892350979 2024-09-15T21:42:56.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:56.082 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:56.082 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182462 2024-09-15T21:42:56.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:42:56.083 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182462 2024-09-15T21:42:56.084 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:56.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182462 2024-09-15T21:42:56.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182462' 2024-09-15T21:42:56.085 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.4 seq 107374182462 2024-09-15T21:42:56.086 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:42:56.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182462 -lt 107374182462 2024-09-15T21:42:56.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:56.516 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:56.516 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018940 2024-09-15T21:42:56.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:42:56.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:56.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018940 2024-09-15T21:42:56.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018940 2024-09-15T21:42:56.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018940' 2024-09-15T21:42:56.519 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.5 seq 128849018940 2024-09-15T21:42:56.519 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:42:56.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018941 -lt 128849018940 2024-09-15T21:42:56.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:56.996 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855412 2024-09-15T21:42:56.996 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:56.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:42:56.999 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855412 2024-09-15T21:42:56.999 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:57.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855412 2024-09-15T21:42:57.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-09-15T21:42:57.000 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.6 seq 150323855412 2024-09-15T21:42:57.001 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:42:57.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855412 -lt 150323855412 2024-09-15T21:42:57.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:57.488 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:57.488 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691879 2024-09-15T21:42:57.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:42:57.490 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691879 2024-09-15T21:42:57.490 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:57.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691879 2024-09-15T21:42:57.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-09-15T21:42:57.492 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.7 seq 171798691879 2024-09-15T21:42:57.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:42:57.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691880 -lt 171798691879 2024-09-15T21:42:57.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:57.955 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528356 2024-09-15T21:42:57.955 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:57.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:42:57.957 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528356 2024-09-15T21:42:57.957 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:57.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528356 2024-09-15T21:42:57.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2024-09-15T21:42:57.958 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.8 seq 193273528356 2024-09-15T21:42:57.959 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:42:58.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528356 -lt 193273528356 2024-09-15T21:42:58.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:58.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:58.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364827 2024-09-15T21:42:58.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:42:58.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364827 2024-09-15T21:42:58.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:58.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364827 2024-09-15T21:42:58.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364827' 2024-09-15T21:42:58.423 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.9 seq 214748364827 2024-09-15T21:42:58.423 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:42:58.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364827 -lt 214748364827 2024-09-15T21:42:58.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:58.900 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201302 2024-09-15T21:42:58.900 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:58.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-15T21:42:58.902 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201302 2024-09-15T21:42:58.902 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:58.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201302 2024-09-15T21:42:58.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201302' 2024-09-15T21:42:58.904 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.10 seq 236223201302 2024-09-15T21:42:58.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T21:42:59.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201303 -lt 236223201302 2024-09-15T21:42:59.377 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:42:59.378 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:42:59.378 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:42:59.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-15T21:42:59.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:42:59.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:42:59.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:42:59.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:42:59.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:42:59.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:42:59.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:43:00.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-15T21:43:00.368 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:43:00.369 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:43:00.369 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:43:00.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-15T21:43:00.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:43:00.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:43:00.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-15T21:43:00.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDecpool 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:43:00.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:43:00.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:43:01.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:43:01.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-15T21:43:01.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-15T21:43:01.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:43:01.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-15T21:43:01.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:43:01.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2024-09-15T21:43:01.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2024-09-15T21:43:02.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2024-09-15T21:43:02.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-15T21:43:02.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:43:02.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-15T21:43:02.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:43:02.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/5 2024-09-15T21:43:02.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:43:02.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:43:02.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:43:02.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:43:02.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-15T21:43:02.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-15T21:43:02.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:43:02.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:43:02.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:43:02.224 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:43:02.224 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:43:02.224 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:43:02.224 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:43:02.224 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:02.224 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:43:02.224 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:43:02.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:43:02.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:43:02.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:43:02.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:43:02.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:43:02.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:43:02.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:43:02.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:43:02.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:43:02.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:43:02.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/5 2024-09-15T21:43:02.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-15T21:43:02.226 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.5 2024-09-15T21:43:02.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:43:02.226 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/5/whoami 2024-09-15T21:43:02.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-15T21:43:02.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:43:02.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:43:02.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:43:02.260 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:02.256+0000 7fee984e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:02.440 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:02.436+0000 7fee984e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:02.441 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:02.436+0000 7fee984e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:02.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-15T21:43:02.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:43:02.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:43:02.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:43:02.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:43:02.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:02.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:43:02.845 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:43:02.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:02.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:43:03.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:03.755 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:03.748+0000 7fee984e37c0 -1 Falling back to public interface 2024-09-15T21:43:04.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:04.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:04.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:43:04.666 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:43:04.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:04.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:43:05.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:06.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:06.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:06.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:43:06.097 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:43:06.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:06.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:43:06.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:07.047 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:07.040+0000 7fee984e37c0 -1 osd.5 83 log_to_monitors true 2024-09-15T21:43:07.531 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:07.531 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:07.531 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:43:07.532 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:43:07.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:07.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:43:08.530 INFO:tasks.workunit.client.0.smithi104.stderr:osd.5 up in weight 1 up_from 88 up_thru 88 down_at 84 last_clean_interval [30,83) [v2:127.0.0.1:6842/1323096505,v1:127.0.0.1:6843/1323096505] [v2:127.0.0.1:6844/1323096505,v1:127.0.0.1:6845/1323096505] exists,up a5825e7b-74c1-47c2-abb3-f77a8dbdcad5 2024-09-15T21:43:08.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:43:08.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:43:08.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:43:08.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:43:08.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:43:08.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:43:08.531 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:43:08.531 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:43:08.532 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:43:08.532 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:43:08.532 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:43:08.532 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:43:08.532 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:43:08.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:43:08.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:43:08.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:43:08.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:43:08.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:43:08.632 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:43:09.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:43:09.589 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:43:09.589 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:43:09.589 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:43:09.589 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:43:09.589 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:43:09.589 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:43:09.589 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:43:09.589 INFO:tasks.workunit.client.0.smithi104.stderr:8 2024-09-15T21:43:09.589 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:43:09.589 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:43:09.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:43:09.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:09.590 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:43:09.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836577 2024-09-15T21:43:09.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836577 2024-09-15T21:43:09.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577' 2024-09-15T21:43:09.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:09.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:43:10.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673050 2024-09-15T21:43:10.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673050 2024-09-15T21:43:10.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050' 2024-09-15T21:43:10.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:10.309 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:43:10.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509525 2024-09-15T21:43:10.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509525 2024-09-15T21:43:10.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509525' 2024-09-15T21:43:10.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:10.473 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:43:10.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350984 2024-09-15T21:43:10.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350984 2024-09-15T21:43:10.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509525 3-347892350984' 2024-09-15T21:43:10.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:10.672 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:43:10.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182467 2024-09-15T21:43:10.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182467 2024-09-15T21:43:10.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509525 3-347892350984 4-107374182467' 2024-09-15T21:43:10.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:10.828 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:43:10.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122051 2024-09-15T21:43:10.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122051 2024-09-15T21:43:10.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509525 3-347892350984 4-107374182467 5-377957122051' 2024-09-15T21:43:10.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:10.981 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:43:11.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855417 2024-09-15T21:43:11.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855417 2024-09-15T21:43:11.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509525 3-347892350984 4-107374182467 5-377957122051 6-150323855417' 2024-09-15T21:43:11.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:11.124 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:43:11.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691885 2024-09-15T21:43:11.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691885 2024-09-15T21:43:11.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509525 3-347892350984 4-107374182467 5-377957122051 6-150323855417 7-171798691885' 2024-09-15T21:43:11.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:11.264 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:43:11.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528361 2024-09-15T21:43:11.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528361 2024-09-15T21:43:11.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509525 3-347892350984 4-107374182467 5-377957122051 6-150323855417 7-171798691885 8-193273528361' 2024-09-15T21:43:11.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:11.415 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:43:11.566 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364832 2024-09-15T21:43:11.566 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364832 2024-09-15T21:43:11.566 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509525 3-347892350984 4-107374182467 5-377957122051 6-150323855417 7-171798691885 8-193273528361 9-214748364832' 2024-09-15T21:43:11.566 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:11.566 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T21:43:11.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201307 2024-09-15T21:43:11.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201307 2024-09-15T21:43:11.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509525 3-347892350984 4-107374182467 5-377957122051 6-150323855417 7-171798691885 8-193273528361 9-214748364832 10-236223201307' 2024-09-15T21:43:11.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:11.718 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836577 2024-09-15T21:43:11.719 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:11.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:43:11.720 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836577 2024-09-15T21:43:11.720 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:11.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836577 2024-09-15T21:43:11.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-09-15T21:43:11.725 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 21474836577 2024-09-15T21:43:11.725 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:43:12.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836578 -lt 21474836577 2024-09-15T21:43:12.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:12.192 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673050 2024-09-15T21:43:12.192 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:12.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:43:12.194 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673050 2024-09-15T21:43:12.194 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:12.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673050 2024-09-15T21:43:12.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-09-15T21:43:12.195 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 42949673050 2024-09-15T21:43:12.195 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:43:12.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673050 -lt 42949673050 2024-09-15T21:43:12.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:12.668 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509525 2024-09-15T21:43:12.668 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:12.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:43:12.670 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509525 2024-09-15T21:43:12.670 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:12.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509525 2024-09-15T21:43:12.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-09-15T21:43:12.672 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 64424509525 2024-09-15T21:43:12.672 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:43:13.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509525 -lt 64424509525 2024-09-15T21:43:13.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:13.111 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892350984 2024-09-15T21:43:13.111 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:13.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:43:13.113 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892350984 2024-09-15T21:43:13.113 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:13.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350984 2024-09-15T21:43:13.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892350984' 2024-09-15T21:43:13.114 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 347892350984 2024-09-15T21:43:13.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:43:13.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350985 -lt 347892350984 2024-09-15T21:43:13.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:13.523 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182467 2024-09-15T21:43:13.523 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:13.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:43:13.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182467 2024-09-15T21:43:13.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:13.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182467 2024-09-15T21:43:13.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-09-15T21:43:13.526 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.4 seq 107374182467 2024-09-15T21:43:13.526 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:43:13.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182468 -lt 107374182467 2024-09-15T21:43:13.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:13.947 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122051 2024-09-15T21:43:13.947 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:13.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:43:13.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122051 2024-09-15T21:43:13.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:13.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122051 2024-09-15T21:43:13.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122051' 2024-09-15T21:43:13.951 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.5 seq 377957122051 2024-09-15T21:43:13.951 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:43:14.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122052 -lt 377957122051 2024-09-15T21:43:14.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:14.387 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855417 2024-09-15T21:43:14.387 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:14.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:43:14.388 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855417 2024-09-15T21:43:14.389 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:14.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855417 2024-09-15T21:43:14.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855417' 2024-09-15T21:43:14.391 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.6 seq 150323855417 2024-09-15T21:43:14.391 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:43:14.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855418 -lt 150323855417 2024-09-15T21:43:14.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:14.809 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:14.809 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691885 2024-09-15T21:43:14.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:43:14.811 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691885 2024-09-15T21:43:14.811 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:14.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691885 2024-09-15T21:43:14.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691885' 2024-09-15T21:43:14.813 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.7 seq 171798691885 2024-09-15T21:43:14.813 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:43:15.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691885 -lt 171798691885 2024-09-15T21:43:15.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:15.224 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528361 2024-09-15T21:43:15.224 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:15.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:43:15.226 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528361 2024-09-15T21:43:15.226 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:15.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528361 2024-09-15T21:43:15.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-09-15T21:43:15.227 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.8 seq 193273528361 2024-09-15T21:43:15.227 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:43:15.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528362 -lt 193273528361 2024-09-15T21:43:15.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:15.680 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364832 2024-09-15T21:43:15.680 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:15.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:43:15.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364832 2024-09-15T21:43:15.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:15.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364832 2024-09-15T21:43:15.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364832' 2024-09-15T21:43:15.683 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.9 seq 214748364832 2024-09-15T21:43:15.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:43:16.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364833 -lt 214748364832 2024-09-15T21:43:16.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:16.112 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201307 2024-09-15T21:43:16.112 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:16.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-15T21:43:16.113 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201307 2024-09-15T21:43:16.113 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:16.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201307 2024-09-15T21:43:16.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201307' 2024-09-15T21:43:16.114 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.10 seq 236223201307 2024-09-15T21:43:16.115 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T21:43:16.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201308 -lt 236223201307 2024-09-15T21:43:16.556 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:43:16.557 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:43:16.557 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:43:17.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-15T21:43:17.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:43:17.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:43:17.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:43:17.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:43:17.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:43:17.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:43:17.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:43:17.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-15T21:43:17.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:43:17.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:43:17.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:43:18.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-15T21:43:18.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:43:18.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:43:18.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:313: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2024-09-15T21:43:18.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:318: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2024-09-15T21:43:19.316 INFO:tasks.workunit.client.0.smithi104.stdout:crush-device-class= 2024-09-15T21:43:19.317 INFO:tasks.workunit.client.0.smithi104.stdout:crush-failure-domain=host 2024-09-15T21:43:19.317 INFO:tasks.workunit.client.0.smithi104.stdout:crush-num-failure-domains=0 2024-09-15T21:43:19.317 INFO:tasks.workunit.client.0.smithi104.stdout:crush-osds-per-failure-domain=0 2024-09-15T21:43:19.317 INFO:tasks.workunit.client.0.smithi104.stdout:crush-root=default 2024-09-15T21:43:19.317 INFO:tasks.workunit.client.0.smithi104.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2024-09-15T21:43:19.317 INFO:tasks.workunit.client.0.smithi104.stdout:k=-1 2024-09-15T21:43:19.321 INFO:tasks.workunit.client.0.smithi104.stdout:l=-1 2024-09-15T21:43:19.321 INFO:tasks.workunit.client.0.smithi104.stdout:layers=[ [ "cDD", "" ] ] 2024-09-15T21:43:19.321 INFO:tasks.workunit.client.0.smithi104.stdout:m=-1 2024-09-15T21:43:19.321 INFO:tasks.workunit.client.0.smithi104.stdout:mapping=_DD 2024-09-15T21:43:19.321 INFO:tasks.workunit.client.0.smithi104.stdout:plugin=lrc 2024-09-15T21:43:19.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:319: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2024-09-15T21:43:19.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2024-09-15T21:43:20.159 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'remap-pool' already exists 2024-09-15T21:43:20.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:43:21.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:327: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2024-09-15T21:43:21.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-15T21:43:21.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=remap-pool 2024-09-15T21:43:21.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=1 2024-09-15T21:43:21.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=2 2024-09-15T21:43:21.181 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-15T21:43:21.182 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-15T21:43:21.204 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-15T21:43:21.204 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2024-09-15T21:43:21.205 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-15T21:43:21.206 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-15T21:43:21.227 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-15T21:43:21.227 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2024-09-15T21:43:21.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2024-09-15T21:43:21.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2024-09-15T21:43:21.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2024-09-15T21:43:22.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2024-09-15T21:43:22.989 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-15T21:43:22.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=remap-pool 2024-09-15T21:43:22.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGremap-pool 2024-09-15T21:43:22.990 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-15T21:43:22.990 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:43:23.511 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-15T21:43:23.511 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:43:23.511 INFO:tasks.workunit.client.0.smithi104.stderr:9' 2024-09-15T21:43:23.511 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 6 9 2024-09-15T21:43:23.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('4' '6' '9') 2024-09-15T21:43:23.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-09-15T21:43:23.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-09-15T21:43:23.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-15T21:43:23.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2024-09-15T21:43:23.639 INFO:tasks.workunit.client.0.smithi104.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T21:43:23.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-15T21:43:23.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-15T21:43:23.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2024-09-15T21:43:23.770 INFO:tasks.workunit.client.0.smithi104.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T21:43:23.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-15T21:43:23.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-15T21:43:23.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-15T21:43:23.899 INFO:tasks.workunit.client.0.smithi104.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T21:43:23.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-15T21:43:23.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-15T21:43:23.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:43:23.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-15T21:43:23.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-15T21:43:23.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=remap-pool 2024-09-15T21:43:23.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGremap-pool 2024-09-15T21:43:23.904 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-15T21:43:23.904 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:43:24.375 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-15T21:43:24.375 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:43:24.375 INFO:tasks.workunit.client.0.smithi104.stderr:9' 2024-09-15T21:43:24.375 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 6 9 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('4' '6' '9') 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=6 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=6 2024-09-15T21:43:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:43:24.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2024-09-15T21:43:24.377 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:43:24.377 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:43:24.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:43:24.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:43:24.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:43:25.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:43:25.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-15T21:43:25.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-15T21:43:25.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:43:25.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=6 2024-09-15T21:43:25.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:43:25.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2024-09-15T21:43:25.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2024-09-15T21:43:26.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2024-09-15T21:43:26.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-15T21:43:26.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:43:26.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=6 2024-09-15T21:43:26.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:43:26.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/6 2024-09-15T21:43:26.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:43:26.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:43:26.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:43:26.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:43:26.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-15T21:43:26.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-15T21:43:26.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:43:26.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:43:26.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:43:26.870 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:43:26.870 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:43:26.870 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:43:26.872 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:43:26.872 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:26.872 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:43:26.872 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:43:26.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:43:26.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:43:26.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:43:26.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:43:26.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:43:26.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:43:26.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:43:26.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:43:26.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:43:26.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:43:26.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/6 2024-09-15T21:43:26.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.6 2024-09-15T21:43:26.873 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.6 2024-09-15T21:43:26.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 6 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:43:26.875 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/6/whoami 2024-09-15T21:43:26.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 6 = 6 ']' 2024-09-15T21:43:26.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:43:26.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:43:26.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:43:26.907 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:26.900+0000 7f42d9bc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:27.065 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:27.060+0000 7f42d9bc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:27.066 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:27.060+0000 7f42d9bc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:27.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 6 2024-09-15T21:43:27.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:43:27.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-15T21:43:27.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:43:27.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:43:27.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:27.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:43:27.395 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:43:27.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:27.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:43:27.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:28.338 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:28.332+0000 7f42d9bc27c0 -1 Falling back to public interface 2024-09-15T21:43:28.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:28.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:28.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:43:28.825 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:43:28.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:28.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:43:29.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:30.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:30.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:30.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:43:30.333 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:43:30.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:30.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:43:30.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:31.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:31.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:31.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:43:31.809 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:43:31.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:31.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:43:31.951 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:31.944+0000 7f42d9bc27c0 -1 osd.6 100 log_to_monitors true 2024-09-15T21:43:32.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:33.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:33.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:33.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:43:33.283 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:43:33.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:33.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:43:34.637 INFO:tasks.workunit.client.0.smithi104.stderr:osd.6 up in weight 1 up_from 106 up_thru 106 down_at 102 last_clean_interval [35,100) [v2:127.0.0.1:6850/2386685813,v1:127.0.0.1:6851/2386685813] [v2:127.0.0.1:6852/2386685813,v1:127.0.0.1:6853/2386685813] exists,up 8f04fbd0-c08f-4b32-bb5e-b3d10aa3bc9b 2024-09-15T21:43:34.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:43:34.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:43:34.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:43:34.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:43:34.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:43:34.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:43:34.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:43:34.638 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:43:34.638 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:43:34.638 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:43:34.639 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:43:34.639 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:43:34.639 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:43:34.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:43:34.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:43:34.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:43:34.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:43:34.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:43:34.746 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:43:35.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:43:35.206 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:43:35.206 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:43:35.206 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:43:35.206 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:43:35.206 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:43:35.206 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:43:35.207 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:43:35.207 INFO:tasks.workunit.client.0.smithi104.stderr:8 2024-09-15T21:43:35.207 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:43:35.207 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:43:35.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:43:35.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:35.207 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:43:35.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836585 2024-09-15T21:43:35.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836585 2024-09-15T21:43:35.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585' 2024-09-15T21:43:35.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:35.647 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:43:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673057 2024-09-15T21:43:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673057 2024-09-15T21:43:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673057' 2024-09-15T21:43:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:35.854 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:43:36.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509533 2024-09-15T21:43:36.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509533 2024-09-15T21:43:36.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673057 2-64424509533' 2024-09-15T21:43:36.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:36.234 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:43:36.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350992 2024-09-15T21:43:36.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350992 2024-09-15T21:43:36.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673057 2-64424509533 3-347892350992' 2024-09-15T21:43:36.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:36.389 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:43:36.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182474 2024-09-15T21:43:36.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182474 2024-09-15T21:43:36.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673057 2-64424509533 3-347892350992 4-107374182474' 2024-09-15T21:43:36.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:36.540 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:43:36.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122058 2024-09-15T21:43:36.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122058 2024-09-15T21:43:36.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673057 2-64424509533 3-347892350992 4-107374182474 5-377957122058' 2024-09-15T21:43:36.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:36.693 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:43:36.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533379 2024-09-15T21:43:36.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533379 2024-09-15T21:43:36.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673057 2-64424509533 3-347892350992 4-107374182474 5-377957122058 6-455266533379' 2024-09-15T21:43:36.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:36.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:43:36.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691892 2024-09-15T21:43:36.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691892 2024-09-15T21:43:36.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673057 2-64424509533 3-347892350992 4-107374182474 5-377957122058 6-455266533379 7-171798691892' 2024-09-15T21:43:36.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:37.000 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:43:37.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528368 2024-09-15T21:43:37.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528368 2024-09-15T21:43:37.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673057 2-64424509533 3-347892350992 4-107374182474 5-377957122058 6-455266533379 7-171798691892 8-193273528368' 2024-09-15T21:43:37.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:37.158 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:43:37.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364839 2024-09-15T21:43:37.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364839 2024-09-15T21:43:37.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673057 2-64424509533 3-347892350992 4-107374182474 5-377957122058 6-455266533379 7-171798691892 8-193273528368 9-214748364839' 2024-09-15T21:43:37.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:37.348 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T21:43:37.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201314 2024-09-15T21:43:37.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201314 2024-09-15T21:43:37.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673057 2-64424509533 3-347892350992 4-107374182474 5-377957122058 6-455266533379 7-171798691892 8-193273528368 9-214748364839 10-236223201314' 2024-09-15T21:43:37.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:37.500 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836585 2024-09-15T21:43:37.500 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:37.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:43:37.501 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836585 2024-09-15T21:43:37.501 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:37.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836585 2024-09-15T21:43:37.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-09-15T21:43:37.502 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 21474836585 2024-09-15T21:43:37.502 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:43:37.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836585 -lt 21474836585 2024-09-15T21:43:37.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:37.917 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673057 2024-09-15T21:43:37.917 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:37.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:43:37.920 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673057 2024-09-15T21:43:37.920 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:37.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673057 2024-09-15T21:43:37.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-09-15T21:43:37.921 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 42949673057 2024-09-15T21:43:37.921 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:43:38.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673057 -lt 42949673057 2024-09-15T21:43:38.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:38.339 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509533 2024-09-15T21:43:38.339 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:38.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:43:38.341 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509533 2024-09-15T21:43:38.341 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:38.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509533 2024-09-15T21:43:38.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-09-15T21:43:38.343 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 64424509533 2024-09-15T21:43:38.343 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:43:38.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509533 -lt 64424509533 2024-09-15T21:43:38.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:38.785 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892350992 2024-09-15T21:43:38.785 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:38.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:43:38.788 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892350992 2024-09-15T21:43:38.788 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:38.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350992 2024-09-15T21:43:38.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892350992' 2024-09-15T21:43:38.789 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 347892350992 2024-09-15T21:43:38.790 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:43:39.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350992 -lt 347892350992 2024-09-15T21:43:39.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:39.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182474 2024-09-15T21:43:39.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:39.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:43:39.246 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182474 2024-09-15T21:43:39.246 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:39.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182474 2024-09-15T21:43:39.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-09-15T21:43:39.247 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.4 seq 107374182474 2024-09-15T21:43:39.247 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:43:39.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182475 -lt 107374182474 2024-09-15T21:43:39.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:39.740 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122058 2024-09-15T21:43:39.740 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:39.742 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:43:39.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122058 2024-09-15T21:43:39.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:39.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122058 2024-09-15T21:43:39.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122058' 2024-09-15T21:43:39.744 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.5 seq 377957122058 2024-09-15T21:43:39.744 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:43:40.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122059 -lt 377957122058 2024-09-15T21:43:40.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:40.188 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-455266533379 2024-09-15T21:43:40.188 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:40.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:43:40.191 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-455266533379 2024-09-15T21:43:40.191 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:40.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533379 2024-09-15T21:43:40.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 455266533379' 2024-09-15T21:43:40.192 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.6 seq 455266533379 2024-09-15T21:43:40.192 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:43:40.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533380 -lt 455266533379 2024-09-15T21:43:40.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:40.603 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691892 2024-09-15T21:43:40.604 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:40.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:43:40.605 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691892 2024-09-15T21:43:40.605 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:40.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691892 2024-09-15T21:43:40.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-09-15T21:43:40.608 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.7 seq 171798691892 2024-09-15T21:43:40.608 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:43:41.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691892 -lt 171798691892 2024-09-15T21:43:41.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:41.096 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528368 2024-09-15T21:43:41.096 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:41.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:43:41.098 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528368 2024-09-15T21:43:41.098 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:41.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528368 2024-09-15T21:43:41.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528368' 2024-09-15T21:43:41.100 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.8 seq 193273528368 2024-09-15T21:43:41.100 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:43:41.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528369 -lt 193273528368 2024-09-15T21:43:41.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:41.561 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:41.561 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364839 2024-09-15T21:43:41.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:43:41.563 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364839 2024-09-15T21:43:41.563 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:41.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364839 2024-09-15T21:43:41.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364839' 2024-09-15T21:43:41.565 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.9 seq 214748364839 2024-09-15T21:43:41.565 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:43:42.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364840 -lt 214748364839 2024-09-15T21:43:42.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:42.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201314 2024-09-15T21:43:42.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:42.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-15T21:43:42.032 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201314 2024-09-15T21:43:42.032 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:42.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201314 2024-09-15T21:43:42.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201314' 2024-09-15T21:43:42.033 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.10 seq 236223201314 2024-09-15T21:43:42.033 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T21:43:42.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201315 -lt 236223201314 2024-09-15T21:43:42.446 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:43:42.447 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:43:42.447 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:43:43.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-15T21:43:43.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:43:43.017 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:43:43.017 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:43:43.017 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:43:43.017 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:43:43.017 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:43:43.017 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:43:43.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-15T21:43:43.517 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:43:43.517 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:43:43.517 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:43:44.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-15T21:43:44.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:43:44.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:43:44.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-15T21:43:44.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-15T21:43:44.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:43:44.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-15T21:43:44.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:43:44.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-15T21:43:44.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-15T21:43:44.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:43:44.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-15T21:43:44.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:43:44.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2024-09-15T21:43:44.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDremap-pool 2024-09-15T21:43:44.070 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:43:44.070 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:43:44.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:43:44.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:43:44.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:43:44.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:43:44.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-15T21:43:44.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-15T21:43:44.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:43:44.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-15T21:43:44.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:43:44.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2024-09-15T21:43:44.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2024-09-15T21:43:45.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2024-09-15T21:43:45.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-15T21:43:45.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:43:45.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-15T21:43:45.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:43:45.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/9 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:43:45.287 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:43:45.288 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:45.288 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:43:45.289 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:43:45.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' 2024-09-15T21:43:45.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:43:45.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T21:43:45.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T21:43:45.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:43:45.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:43:45.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:43:45.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:43:45.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:43:45.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:43:45.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/9 2024-09-15T21:43:45.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-15T21:43:45.291 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.9 2024-09-15T21:43:45.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=941eece4-8898-4fe2-bab8-717ddbc6806b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.50061/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:43:45.291 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/9/whoami 2024-09-15T21:43:45.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-15T21:43:45.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:43:45.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:43:45.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:43:45.321 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:45.316+0000 7f2c0ff847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:45.323 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:45.320+0000 7f2c0ff847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:45.325 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:45.320+0000 7f2c0ff847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:46.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-15T21:43:46.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:43:46.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:43:46.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:43:46.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:43:46.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:46.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:43:46.258 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:43:46.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:46.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:43:46.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:46.834 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:46.828+0000 7f2c0ff847c0 -1 Falling back to public interface 2024-09-15T21:43:47.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:47.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:47.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:43:47.761 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:43:47.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:47.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:43:48.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:49.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:49.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:49.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:43:49.251 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:43:49.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:49.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:43:49.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:50.139 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:43:50.132+0000 7f2c0ff847c0 -1 osd.9 109 log_to_monitors true 2024-09-15T21:43:50.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:50.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:50.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:43:50.737 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:43:50.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:50.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:43:51.485 INFO:tasks.workunit.client.0.smithi104.stderr:osd.9 up in weight 1 up_from 113 up_thru 70 down_at 110 last_clean_interval [50,109) [v2:127.0.0.1:6874/407589960,v1:127.0.0.1:6875/407589960] [v2:127.0.0.1:6876/407589960,v1:127.0.0.1:6877/407589960] exists,up 86ff2c29-70ab-4d48-bd99-44b68dc774a0 2024-09-15T21:43:51.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:43:51.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:43:51.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:43:51.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:43:51.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:43:51.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:43:51.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:43:51.486 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:43:51.487 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:43:51.487 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:43:51.487 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:43:51.487 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:43:51.487 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:43:51.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:43:51.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:43:51.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:43:51.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:43:51.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:43:51.594 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:43:52.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:43:52.369 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:43:52.369 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:43:52.369 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:43:52.369 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:43:52.370 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:43:52.370 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:43:52.370 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:43:52.370 INFO:tasks.workunit.client.0.smithi104.stderr:8 2024-09-15T21:43:52.370 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:43:52.370 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:43:52.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:43:52.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:52.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:43:52.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836590 2024-09-15T21:43:52.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836590 2024-09-15T21:43:52.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590' 2024-09-15T21:43:52.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:52.666 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:43:52.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673062 2024-09-15T21:43:52.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673062 2024-09-15T21:43:52.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673062' 2024-09-15T21:43:52.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:52.999 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:43:53.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509538 2024-09-15T21:43:53.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509538 2024-09-15T21:43:53.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673062 2-64424509538' 2024-09-15T21:43:53.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:53.160 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:43:53.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350997 2024-09-15T21:43:53.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350997 2024-09-15T21:43:53.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673062 2-64424509538 3-347892350997' 2024-09-15T21:43:53.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:53.630 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:43:53.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182480 2024-09-15T21:43:53.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182480 2024-09-15T21:43:53.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673062 2-64424509538 3-347892350997 4-107374182480' 2024-09-15T21:43:53.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:53.816 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:43:53.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122064 2024-09-15T21:43:53.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122064 2024-09-15T21:43:53.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673062 2-64424509538 3-347892350997 4-107374182480 5-377957122064' 2024-09-15T21:43:53.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:53.969 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:43:54.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533385 2024-09-15T21:43:54.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533385 2024-09-15T21:43:54.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673062 2-64424509538 3-347892350997 4-107374182480 5-377957122064 6-455266533385' 2024-09-15T21:43:54.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:54.121 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:43:54.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691897 2024-09-15T21:43:54.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691897 2024-09-15T21:43:54.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673062 2-64424509538 3-347892350997 4-107374182480 5-377957122064 6-455266533385 7-171798691897' 2024-09-15T21:43:54.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:54.273 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:43:54.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528374 2024-09-15T21:43:54.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528374 2024-09-15T21:43:54.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673062 2-64424509538 3-347892350997 4-107374182480 5-377957122064 6-455266533385 7-171798691897 8-193273528374' 2024-09-15T21:43:54.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:54.425 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:43:54.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=485331304451 2024-09-15T21:43:54.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 485331304451 2024-09-15T21:43:54.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673062 2-64424509538 3-347892350997 4-107374182480 5-377957122064 6-455266533385 7-171798691897 8-193273528374 9-485331304451' 2024-09-15T21:43:54.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:54.575 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T21:43:54.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201320 2024-09-15T21:43:54.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201320 2024-09-15T21:43:54.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673062 2-64424509538 3-347892350997 4-107374182480 5-377957122064 6-455266533385 7-171798691897 8-193273528374 9-485331304451 10-236223201320' 2024-09-15T21:43:54.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:54.716 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836590 2024-09-15T21:43:54.716 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:54.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:43:54.717 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836590 2024-09-15T21:43:54.717 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:54.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836590 2024-09-15T21:43:54.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2024-09-15T21:43:54.719 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 21474836590 2024-09-15T21:43:54.719 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:43:55.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836590 -lt 21474836590 2024-09-15T21:43:55.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:55.226 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673062 2024-09-15T21:43:55.226 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:55.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:43:55.228 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673062 2024-09-15T21:43:55.228 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:55.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673062 2024-09-15T21:43:55.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-09-15T21:43:55.229 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 42949673062 2024-09-15T21:43:55.230 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:43:55.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673063 -lt 42949673062 2024-09-15T21:43:55.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:55.671 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509538 2024-09-15T21:43:55.671 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:55.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:43:55.673 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509538 2024-09-15T21:43:55.673 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:55.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509538 2024-09-15T21:43:55.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509538' 2024-09-15T21:43:55.675 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 64424509538 2024-09-15T21:43:55.675 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:43:56.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509538 -lt 64424509538 2024-09-15T21:43:56.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:56.161 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:56.161 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892350997 2024-09-15T21:43:56.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:43:56.162 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892350997 2024-09-15T21:43:56.162 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:56.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350997 2024-09-15T21:43:56.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892350997' 2024-09-15T21:43:56.164 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 347892350997 2024-09-15T21:43:56.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:43:56.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350997 -lt 347892350997 2024-09-15T21:43:56.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:56.596 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:56.596 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182480 2024-09-15T21:43:56.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:43:56.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182480 2024-09-15T21:43:56.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:56.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182480 2024-09-15T21:43:56.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-09-15T21:43:56.600 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.4 seq 107374182480 2024-09-15T21:43:56.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:43:57.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182480 -lt 107374182480 2024-09-15T21:43:57.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:57.033 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122064 2024-09-15T21:43:57.033 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:57.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:43:57.034 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122064 2024-09-15T21:43:57.035 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:57.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122064 2024-09-15T21:43:57.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122064' 2024-09-15T21:43:57.036 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.5 seq 377957122064 2024-09-15T21:43:57.036 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:43:57.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122064 -lt 377957122064 2024-09-15T21:43:57.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:57.476 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-455266533385 2024-09-15T21:43:57.476 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:57.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:43:57.478 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-455266533385 2024-09-15T21:43:57.478 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:57.479 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533385 2024-09-15T21:43:57.479 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 455266533385' 2024-09-15T21:43:57.479 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.6 seq 455266533385 2024-09-15T21:43:57.479 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:43:57.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533385 -lt 455266533385 2024-09-15T21:43:57.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:57.980 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691897 2024-09-15T21:43:57.980 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:57.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:43:57.982 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691897 2024-09-15T21:43:57.982 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:57.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691897 2024-09-15T21:43:57.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-09-15T21:43:57.984 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.7 seq 171798691897 2024-09-15T21:43:57.984 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:43:58.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691898 -lt 171798691897 2024-09-15T21:43:58.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:58.437 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528374 2024-09-15T21:43:58.437 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:58.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:43:58.439 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528374 2024-09-15T21:43:58.440 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:58.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528374 2024-09-15T21:43:58.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-09-15T21:43:58.441 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.8 seq 193273528374 2024-09-15T21:43:58.441 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:43:58.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528374 -lt 193273528374 2024-09-15T21:43:58.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:58.922 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:58.922 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-485331304451 2024-09-15T21:43:58.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:43:58.924 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-485331304451 2024-09-15T21:43:58.924 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:58.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=485331304451 2024-09-15T21:43:58.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 485331304451' 2024-09-15T21:43:58.925 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.9 seq 485331304451 2024-09-15T21:43:58.925 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:43:59.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 485331304452 -lt 485331304451 2024-09-15T21:43:59.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:59.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201320 2024-09-15T21:43:59.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:59.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-15T21:43:59.372 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201320 2024-09-15T21:43:59.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:59.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201320 2024-09-15T21:43:59.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201320' 2024-09-15T21:43:59.374 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.10 seq 236223201320 2024-09-15T21:43:59.374 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T21:43:59.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201321 -lt 236223201320 2024-09-15T21:43:59.843 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:43:59.843 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:43:59.844 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:44:00.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-15T21:44:00.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:44:00.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:44:00.443 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:44:00.443 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:44:00.443 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:44:00.443 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:44:00.443 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:44:00.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-15T21:44:00.879 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:44:00.879 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:44:00.879 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:44:01.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-15T21:44:01.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:44:01.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:44:01.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:329: TEST_chunk_mapping: delete_pool remap-pool 2024-09-15T21:44:01.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=remap-pool 2024-09-15T21:44:01.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2024-09-15T21:44:01.991 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'remap-pool' does not exist 2024-09-15T21:44:02.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:330: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2024-09-15T21:44:03.008 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile remap-profile does not exist 2024-09-15T21:44:03.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T21:44:03.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2024-09-15T21:44:03.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2024-09-15T21:44:03.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: erasure_code_plugin_exists: local plugin=isa 2024-09-15T21:44:03.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: local status 2024-09-15T21:44:03.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local grepstr 2024-09-15T21:44:03.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: local s 2024-09-15T21:44:03.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: case `uname` in 2024-09-15T21:44:03.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: uname 2024-09-15T21:44:03.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: erasure_code_plugin_exists: grepstr='isa.*No such file' 2024-09-15T21:44:03.030 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2024-09-15T21:44:04.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: s= 2024-09-15T21:44:04.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local status=0 2024-09-15T21:44:04.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-15T21:44:04.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-15T21:44:04.623 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-15T21:44:04.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: return 0 2024-09-15T21:44:04.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:197: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2024-09-15T21:44:04.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:198: TEST_rados_put_get_isa: local poolname=pool-isa 2024-09-15T21:44:04.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:200: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2024-09-15T21:44:05.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:203: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2024-09-15T21:44:05.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2024-09-15T21:44:06.611 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-isa' already exists 2024-09-15T21:44:06.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:44:07.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:206: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2024-09-15T21:44:07.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T21:44:07.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2024-09-15T21:44:07.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T21:44:07.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:07.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T21:44:07.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:07.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T21:44:07.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:07.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T21:44:07.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:07.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T21:44:07.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T21:44:09.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:44:09.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:44:09.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T21:44:09.633 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2024-09-15T21:44:09.633 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-isa 2024-09-15T21:44:09.633 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:44:09.634 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:44:09.634 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-15T21:44:10.105 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-15T21:44:10.105 INFO:tasks.workunit.client.0.smithi104.stderr:10 2024-09-15T21:44:10.105 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:44:10.105 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:44:10.105 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:44:10.105 INFO:tasks.workunit.client.0.smithi104.stderr:8 2024-09-15T21:44:10.105 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:44:10.106 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:44:10.106 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:44:10.106 INFO:tasks.workunit.client.0.smithi104.stderr:9' 2024-09-15T21:44:10.106 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2024-09-15T21:44:10.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('4' '10' '0' '7' '6' '8' '2' '1' '5' '9') 2024-09-15T21:44:10.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T21:44:10.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2024-09-15T21:44:10.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-09-15T21:44:11.579 INFO:tasks.workunit.client.0.smithi104.stderr:osd.9 is already out. 2024-09-15T21:44:11.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-15T21:44:16.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-isa SOMETHING 2024-09-15T21:44:16.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-isa 2024-09-15T21:44:16.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:44:16.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-09-15T21:44:16.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-15T21:44:16.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:44:17.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-15T21:44:17.094 INFO:tasks.workunit.client.0.smithi104.stderr:10 2024-09-15T21:44:17.094 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:44:17.094 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:44:17.095 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:44:17.095 INFO:tasks.workunit.client.0.smithi104.stderr:8 2024-09-15T21:44:17.095 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:44:17.095 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:44:17.095 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:44:17.095 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T21:44:17.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2024-09-15T21:44:17.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:44:17.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:44:17.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-09-15T21:44:17.925 INFO:tasks.workunit.client.0.smithi104.stderr:osd.9 is already in. 2024-09-15T21:44:17.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T21:44:17.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_isa: delete_pool pool-isa 2024-09-15T21:44:17.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-isa 2024-09-15T21:44:17.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2024-09-15T21:44:19.524 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-isa' does not exist 2024-09-15T21:44:19.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T21:44:19.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2024-09-15T21:44:19.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2024-09-15T21:44:19.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:214: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T21:44:19.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T21:44:20.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:44:20.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:44:20.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T21:44:21.008 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2024-09-15T21:44:21.008 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:44:21.008 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:44:21.008 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:44:21.008 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:44:21.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=6 2024-09-15T21:44:21.492 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:44:21.492 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:44:21.492 INFO:tasks.workunit.client.0.smithi104.stderr:9' 2024-09-15T21:44:21.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 6 0 7 9 2024-09-15T21:44:21.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('6' '0' '7' '9') 2024-09-15T21:44:21.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T21:44:21.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2024-09-15T21:44:21.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-09-15T21:44:22.040 INFO:tasks.workunit.client.0.smithi104.stderr:osd.9 is already out. 2024-09-15T21:44:22.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-15T21:44:27.058 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds ecpool SOMETHING 2024-09-15T21:44:27.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:44:27.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:44:27.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-09-15T21:44:27.059 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:44:27.059 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:44:27.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=6 2024-09-15T21:44:27.499 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:44:27.499 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:44:27.499 INFO:tasks.workunit.client.0.smithi104.stderr:5' 2024-09-15T21:44:27.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 6 0 7 5 2024-09-15T21:44:27.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:44:27.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:44:27.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-09-15T21:44:28.059 INFO:tasks.workunit.client.0.smithi104.stderr:osd.9 is already in. 2024-09-15T21:44:28.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T21:44:28.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:216: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2024-09-15T21:44:28.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:217: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2024-09-15T21:44:28.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2024-09-15T21:44:30.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2024-09-15T21:44:30.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2024-09-15T21:44:31.717 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T21:44:31.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:44:32.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2024-09-15T21:44:32.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T21:44:32.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2024-09-15T21:44:32.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T21:44:32.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:32.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T21:44:32.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:32.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T21:44:32.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:32.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T21:44:32.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:44:32.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T21:44:32.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T21:44:34.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:44:35.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:44:35.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T21:44:35.116 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-15T21:44:35.117 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T21:44:35.117 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:44:35.118 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T21:44:35.118 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:44:35.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-15T21:44:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:44:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:44:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:44:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:44:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:44:35.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2024-09-15T21:44:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '1' '9' '4' '3' '10') 2024-09-15T21:44:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T21:44:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2024-09-15T21:44:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2024-09-15T21:44:37.826 INFO:tasks.workunit.client.0.smithi104.stderr:osd.10 is already out. 2024-09-15T21:44:37.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-15T21:44:42.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-15T21:44:42.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T21:44:42.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<10\>' 2024-09-15T21:44:42.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:44:42.851 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T21:44:42.851 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:44:43.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-15T21:44:43.343 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:44:43.343 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:44:43.343 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:44:43.343 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:44:43.343 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T21:44:43.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 6 2024-09-15T21:44:43.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:44:43.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:44:43.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 10 2024-09-15T21:44:43.924 INFO:tasks.workunit.client.0.smithi104.stderr:osd.10 is already in. 2024-09-15T21:44:43.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T21:44:43.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:227: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2024-09-15T21:44:43.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:104: rados_osds_out_in: local dir=td/test-erasure-code 2024-09-15T21:44:43.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: local poolname=pool-jerasure 2024-09-15T21:44:43.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: local objname=SOMETHING 2024-09-15T21:44:43.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-15T21:44:43.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 FFFF 2024-09-15T21:44:43.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-15T21:44:43.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 GGGG 2024-09-15T21:44:43.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-15T21:44:43.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 HHHH 2024-09-15T21:44:43.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-15T21:44:43.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 IIII 2024-09-15T21:44:43.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:116: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T21:44:46.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:117: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:44:46.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:118: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:44:46.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:119: rados_osds_out_in: rm td/test-erasure-code/COPY 2024-09-15T21:44:46.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: wait_for_clean 2024-09-15T21:44:46.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:44:46.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:44:46.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:44:46.716 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:44:46.717 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:44:46.717 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:44:46.717 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:44:46.717 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:44:46.717 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:44:46.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:44:46.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:44:46.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:44:46.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:44:46.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:44:46.825 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:44:47.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:44:47.374 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:44:47.374 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:44:47.374 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:44:47.374 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:44:47.374 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:44:47.374 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:44:47.374 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:44:47.374 INFO:tasks.workunit.client.0.smithi104.stderr:8 2024-09-15T21:44:47.374 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:44:47.375 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:44:47.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:44:47.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:47.375 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:44:47.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836603 2024-09-15T21:44:47.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836603 2024-09-15T21:44:47.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603' 2024-09-15T21:44:47.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:47.603 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:44:47.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673075 2024-09-15T21:44:47.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673075 2024-09-15T21:44:47.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673075' 2024-09-15T21:44:47.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:47.753 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:44:47.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509551 2024-09-15T21:44:47.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509551 2024-09-15T21:44:47.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673075 2-64424509551' 2024-09-15T21:44:47.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:47.955 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:44:48.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892351010 2024-09-15T21:44:48.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892351010 2024-09-15T21:44:48.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673075 2-64424509551 3-347892351010' 2024-09-15T21:44:48.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:48.108 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:44:48.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182493 2024-09-15T21:44:48.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182493 2024-09-15T21:44:48.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673075 2-64424509551 3-347892351010 4-107374182493' 2024-09-15T21:44:48.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:48.296 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:44:48.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122077 2024-09-15T21:44:48.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122077 2024-09-15T21:44:48.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673075 2-64424509551 3-347892351010 4-107374182493 5-377957122077' 2024-09-15T21:44:48.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:48.452 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:44:48.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533398 2024-09-15T21:44:48.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533398 2024-09-15T21:44:48.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673075 2-64424509551 3-347892351010 4-107374182493 5-377957122077 6-455266533398' 2024-09-15T21:44:48.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:48.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:44:48.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691910 2024-09-15T21:44:48.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691910 2024-09-15T21:44:48.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673075 2-64424509551 3-347892351010 4-107374182493 5-377957122077 6-455266533398 7-171798691910' 2024-09-15T21:44:48.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:48.755 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:44:48.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528387 2024-09-15T21:44:48.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528387 2024-09-15T21:44:48.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673075 2-64424509551 3-347892351010 4-107374182493 5-377957122077 6-455266533398 7-171798691910 8-193273528387' 2024-09-15T21:44:48.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:48.908 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:44:49.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=485331304464 2024-09-15T21:44:49.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 485331304464 2024-09-15T21:44:49.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673075 2-64424509551 3-347892351010 4-107374182493 5-377957122077 6-455266533398 7-171798691910 8-193273528387 9-485331304464' 2024-09-15T21:44:49.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:49.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T21:44:49.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201333 2024-09-15T21:44:49.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201333 2024-09-15T21:44:49.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673075 2-64424509551 3-347892351010 4-107374182493 5-377957122077 6-455266533398 7-171798691910 8-193273528387 9-485331304464 10-236223201333' 2024-09-15T21:44:49.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:44:49.231 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836603 2024-09-15T21:44:49.231 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:44:49.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:44:49.232 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836603 2024-09-15T21:44:49.232 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:44:49.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836603 2024-09-15T21:44:49.233 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 21474836603 2024-09-15T21:44:49.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836603' 2024-09-15T21:44:49.234 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:44:49.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836603 -lt 21474836603 2024-09-15T21:44:49.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:44:49.665 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673075 2024-09-15T21:44:49.665 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:44:49.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:44:49.667 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673075 2024-09-15T21:44:49.667 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:44:49.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673075 2024-09-15T21:44:49.669 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 42949673075 2024-09-15T21:44:49.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673075' 2024-09-15T21:44:49.669 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:44:50.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673076 -lt 42949673075 2024-09-15T21:44:50.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:44:50.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509551 2024-09-15T21:44:50.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:44:50.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:44:50.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509551 2024-09-15T21:44:50.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:44:50.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509551 2024-09-15T21:44:50.143 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 64424509551 2024-09-15T21:44:50.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509551' 2024-09-15T21:44:50.143 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:44:50.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509551 -lt 64424509551 2024-09-15T21:44:50.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:44:50.573 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892351010 2024-09-15T21:44:50.573 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:44:50.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:44:50.576 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892351010 2024-09-15T21:44:50.576 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:44:50.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892351010 2024-09-15T21:44:50.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892351010' 2024-09-15T21:44:50.577 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 347892351010 2024-09-15T21:44:50.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:44:51.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892351010 -lt 347892351010 2024-09-15T21:44:51.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:44:51.080 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:44:51.080 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182493 2024-09-15T21:44:51.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:44:51.082 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:44:51.082 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182493 2024-09-15T21:44:51.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182493 2024-09-15T21:44:51.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182493' 2024-09-15T21:44:51.083 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 107374182493 2024-09-15T21:44:51.083 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:44:51.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182493 -lt 107374182493 2024-09-15T21:44:51.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:44:51.506 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122077 2024-09-15T21:44:51.506 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:44:51.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:44:51.508 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122077 2024-09-15T21:44:51.509 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:44:51.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122077 2024-09-15T21:44:51.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122077' 2024-09-15T21:44:51.510 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 377957122077 2024-09-15T21:44:51.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:44:51.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122077 -lt 377957122077 2024-09-15T21:44:51.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:44:51.968 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-455266533398 2024-09-15T21:44:51.968 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:44:51.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:44:51.971 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-455266533398 2024-09-15T21:44:51.971 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:44:51.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533398 2024-09-15T21:44:51.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 455266533398' 2024-09-15T21:44:51.973 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 455266533398 2024-09-15T21:44:51.973 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:44:52.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533398 -lt 455266533398 2024-09-15T21:44:52.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:44:52.405 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691910 2024-09-15T21:44:52.405 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:44:52.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:44:52.407 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691910 2024-09-15T21:44:52.407 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:44:52.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691910 2024-09-15T21:44:52.409 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.7 seq 171798691910 2024-09-15T21:44:52.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691910' 2024-09-15T21:44:52.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:44:52.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691910 -lt 171798691910 2024-09-15T21:44:52.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:44:52.855 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528387 2024-09-15T21:44:52.855 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:44:52.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:44:52.857 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528387 2024-09-15T21:44:52.857 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:44:52.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528387 2024-09-15T21:44:52.858 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.8 seq 193273528387 2024-09-15T21:44:52.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528387' 2024-09-15T21:44:52.859 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:44:53.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528388 -lt 193273528387 2024-09-15T21:44:53.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:44:53.277 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-485331304464 2024-09-15T21:44:53.278 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:44:53.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:44:53.280 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-485331304464 2024-09-15T21:44:53.280 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:44:53.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=485331304464 2024-09-15T21:44:53.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 485331304464' 2024-09-15T21:44:53.281 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.9 seq 485331304464 2024-09-15T21:44:53.281 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:44:53.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 485331304465 -lt 485331304464 2024-09-15T21:44:53.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:44:53.755 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201333 2024-09-15T21:44:53.755 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:44:53.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-15T21:44:53.757 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201333 2024-09-15T21:44:53.757 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:44:53.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201333 2024-09-15T21:44:53.758 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.10 seq 236223201333 2024-09-15T21:44:53.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201333' 2024-09-15T21:44:53.759 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T21:44:54.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201333 -lt 236223201333 2024-09-15T21:44:54.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:44:54.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:44:54.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:44:54.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-15T21:44:54.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:44:54.841 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:44:54.841 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:44:54.841 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:44:54.841 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:44:54.841 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:44:54.841 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:44:55.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-15T21:44:55.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:44:55.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:44:55.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:44:55.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-15T21:44:55.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:44:55.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:44:55.825 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-15T21:44:55.825 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T21:44:55.825 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:44:55.826 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T21:44:55.826 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:44:56.274 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-15T21:44:56.275 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:44:56.275 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:44:56.275 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:44:56.275 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:44:56.275 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:44:56.275 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2024-09-15T21:44:56.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2024-09-15T21:44:56.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: osds=('5' '1' '9' '4' '3' '10') 2024-09-15T21:44:56.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: local -a osds 2024-09-15T21:44:56.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-09-15T21:44:56.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 5 2024-09-15T21:44:56.793 INFO:tasks.workunit.client.0.smithi104.stderr:osd.5 is already out. 2024-09-15T21:44:56.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-09-15T21:44:56.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 1 2024-09-15T21:44:58.373 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 is already out. 2024-09-15T21:44:58.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: wait_for_clean 2024-09-15T21:44:58.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:44:58.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:44:58.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:44:58.397 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:44:58.397 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:44:58.397 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:44:58.398 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:44:58.398 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:44:58.398 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:44:58.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:44:58.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:44:58.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:44:58.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:44:58.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:44:58.506 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:44:59.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:44:59.430 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:44:59.430 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:44:59.430 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:44:59.430 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:44:59.430 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:44:59.430 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:44:59.430 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:44:59.430 INFO:tasks.workunit.client.0.smithi104.stderr:8 2024-09-15T21:44:59.430 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:44:59.431 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:44:59.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:44:59.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:59.431 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:44:59.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836607 2024-09-15T21:44:59.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836607 2024-09-15T21:44:59.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836607' 2024-09-15T21:44:59.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:44:59.871 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:45:00.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673080 2024-09-15T21:45:00.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673080 2024-09-15T21:45:00.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836607 1-42949673080' 2024-09-15T21:45:00.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:00.180 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:45:00.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509555 2024-09-15T21:45:00.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509555 2024-09-15T21:45:00.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836607 1-42949673080 2-64424509555' 2024-09-15T21:45:00.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:00.613 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:45:00.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892351014 2024-09-15T21:45:00.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892351014 2024-09-15T21:45:00.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836607 1-42949673080 2-64424509555 3-347892351014' 2024-09-15T21:45:00.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:00.825 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:45:01.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182497 2024-09-15T21:45:01.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182497 2024-09-15T21:45:01.208 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836607 1-42949673080 2-64424509555 3-347892351014 4-107374182497' 2024-09-15T21:45:01.208 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:01.208 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:45:01.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122081 2024-09-15T21:45:01.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122081 2024-09-15T21:45:01.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836607 1-42949673080 2-64424509555 3-347892351014 4-107374182497 5-377957122081' 2024-09-15T21:45:01.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:01.446 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:45:01.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533402 2024-09-15T21:45:01.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533402 2024-09-15T21:45:01.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836607 1-42949673080 2-64424509555 3-347892351014 4-107374182497 5-377957122081 6-455266533402' 2024-09-15T21:45:01.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:01.759 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:45:02.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691915 2024-09-15T21:45:02.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691915 2024-09-15T21:45:02.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836607 1-42949673080 2-64424509555 3-347892351014 4-107374182497 5-377957122081 6-455266533402 7-171798691915' 2024-09-15T21:45:02.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:02.160 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:45:02.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528391 2024-09-15T21:45:02.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528391 2024-09-15T21:45:02.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836607 1-42949673080 2-64424509555 3-347892351014 4-107374182497 5-377957122081 6-455266533402 7-171798691915 8-193273528391' 2024-09-15T21:45:02.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:02.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:45:02.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=485331304469 2024-09-15T21:45:02.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 485331304469 2024-09-15T21:45:02.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836607 1-42949673080 2-64424509555 3-347892351014 4-107374182497 5-377957122081 6-455266533402 7-171798691915 8-193273528391 9-485331304469' 2024-09-15T21:45:02.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:02.476 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T21:45:02.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201337 2024-09-15T21:45:02.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201337 2024-09-15T21:45:02.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836607 1-42949673080 2-64424509555 3-347892351014 4-107374182497 5-377957122081 6-455266533402 7-171798691915 8-193273528391 9-485331304469 10-236223201337' 2024-09-15T21:45:02.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:02.632 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836607 2024-09-15T21:45:02.633 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:02.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:45:02.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836607 2024-09-15T21:45:02.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:02.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836607 2024-09-15T21:45:02.635 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 21474836607 2024-09-15T21:45:02.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2024-09-15T21:45:02.636 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:45:03.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836608 -lt 21474836607 2024-09-15T21:45:03.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:03.064 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673080 2024-09-15T21:45:03.064 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:03.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:45:03.066 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:03.066 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673080 2024-09-15T21:45:03.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673080 2024-09-15T21:45:03.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673080' 2024-09-15T21:45:03.068 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 42949673080 2024-09-15T21:45:03.068 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:45:03.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673080 -lt 42949673080 2024-09-15T21:45:03.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:03.524 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:03.524 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509555 2024-09-15T21:45:03.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:45:03.526 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509555 2024-09-15T21:45:03.526 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:03.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509555 2024-09-15T21:45:03.527 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 64424509555 2024-09-15T21:45:03.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509555' 2024-09-15T21:45:03.528 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:45:03.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509556 -lt 64424509555 2024-09-15T21:45:03.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:03.948 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892351014 2024-09-15T21:45:03.948 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:03.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:45:03.951 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892351014 2024-09-15T21:45:03.951 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892351014 2024-09-15T21:45:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892351014' 2024-09-15T21:45:03.952 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 347892351014 2024-09-15T21:45:03.953 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:45:04.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892351015 -lt 347892351014 2024-09-15T21:45:04.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:04.411 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182497 2024-09-15T21:45:04.411 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:04.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:45:04.414 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182497 2024-09-15T21:45:04.414 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:04.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182497 2024-09-15T21:45:04.415 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 107374182497 2024-09-15T21:45:04.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182497' 2024-09-15T21:45:04.416 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:45:04.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182498 -lt 107374182497 2024-09-15T21:45:04.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:04.872 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122081 2024-09-15T21:45:04.872 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:04.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:45:04.873 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122081 2024-09-15T21:45:04.873 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:04.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122081 2024-09-15T21:45:04.874 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 377957122081 2024-09-15T21:45:04.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122081' 2024-09-15T21:45:04.875 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:45:05.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122082 -lt 377957122081 2024-09-15T21:45:05.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:05.356 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-455266533402 2024-09-15T21:45:05.356 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:05.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:45:05.358 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:05.358 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-455266533402 2024-09-15T21:45:05.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533402 2024-09-15T21:45:05.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 455266533402' 2024-09-15T21:45:05.359 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 455266533402 2024-09-15T21:45:05.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:45:05.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533403 -lt 455266533402 2024-09-15T21:45:05.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:05.771 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691915 2024-09-15T21:45:05.771 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:05.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:45:05.773 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:05.774 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691915 2024-09-15T21:45:05.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691915 2024-09-15T21:45:05.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691915' 2024-09-15T21:45:05.775 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.7 seq 171798691915 2024-09-15T21:45:05.775 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:45:06.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691915 -lt 171798691915 2024-09-15T21:45:06.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:06.207 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528391 2024-09-15T21:45:06.207 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:06.208 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:45:06.209 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528391 2024-09-15T21:45:06.209 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:06.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528391 2024-09-15T21:45:06.211 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.8 seq 193273528391 2024-09-15T21:45:06.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528391' 2024-09-15T21:45:06.211 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:45:06.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528392 -lt 193273528391 2024-09-15T21:45:06.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:06.655 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-485331304469 2024-09-15T21:45:06.655 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:06.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:45:06.658 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-485331304469 2024-09-15T21:45:06.658 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:06.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=485331304469 2024-09-15T21:45:06.659 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.9 seq 485331304469 2024-09-15T21:45:06.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 485331304469' 2024-09-15T21:45:06.659 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:45:07.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 485331304469 -lt 485331304469 2024-09-15T21:45:07.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:07.156 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201337 2024-09-15T21:45:07.156 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:07.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-15T21:45:07.158 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201337 2024-09-15T21:45:07.158 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:07.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201337 2024-09-15T21:45:07.159 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.10 seq 236223201337 2024-09-15T21:45:07.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201337' 2024-09-15T21:45:07.160 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T21:45:07.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201338 -lt 236223201337 2024-09-15T21:45:07.624 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:45:07.624 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:45:07.624 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:45:08.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-15T21:45:08.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:45:08.207 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:45:08.207 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:45:08.207 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:45:08.207 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:45:08.207 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:45:08.207 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:45:08.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-15T21:45:08.894 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:45:08.894 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:45:08.895 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:45:09.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-15T21:45:09.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:45:09.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:45:09.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-09-15T21:45:09.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-15T21:45:09.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T21:45:09.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:45:09.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<5\>' 2024-09-15T21:45:09.444 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T21:45:09.444 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:45:09.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-15T21:45:09.965 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:45:09.965 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:45:09.965 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:45:09.965 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:45:09.965 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:45:09.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 7 9 4 3 10 2024-09-15T21:45:09.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-09-15T21:45:09.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-15T21:45:09.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T21:45:09.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:45:09.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<1\>' 2024-09-15T21:45:09.967 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T21:45:09.967 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:45:10.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-15T21:45:10.409 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:45:10.409 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:45:10.409 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:45:10.410 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:45:10.410 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:45:10.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 7 9 4 3 10 2024-09-15T21:45:10.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:140: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:45:10.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:141: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:45:10.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-09-15T21:45:10.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 5 2024-09-15T21:45:10.995 INFO:tasks.workunit.client.0.smithi104.stderr:osd.5 is already in. 2024-09-15T21:45:11.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-09-15T21:45:11.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 1 2024-09-15T21:45:12.919 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 is already in. 2024-09-15T21:45:12.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:150: rados_osds_out_in: wait_for_clean 2024-09-15T21:45:12.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:45:12.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:45:12.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:45:12.941 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:45:12.941 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:45:12.941 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:45:12.941 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:45:12.941 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:45:12.941 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:45:13.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:45:13.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:45:13.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:45:13.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:45:13.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:45:13.050 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:45:13.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:45:13.699 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:45:13.699 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:45:13.700 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:45:13.700 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:45:13.700 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:45:13.700 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:45:13.700 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T21:45:13.701 INFO:tasks.workunit.client.0.smithi104.stderr:8 2024-09-15T21:45:13.701 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:45:13.701 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:45:13.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:45:13.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:13.701 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:45:14.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836612 2024-09-15T21:45:14.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836612 2024-09-15T21:45:14.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836612' 2024-09-15T21:45:14.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:14.075 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:45:14.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673085 2024-09-15T21:45:14.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673085 2024-09-15T21:45:14.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836612 1-42949673085' 2024-09-15T21:45:14.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:14.329 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:45:14.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509560 2024-09-15T21:45:14.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509560 2024-09-15T21:45:14.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836612 1-42949673085 2-64424509560' 2024-09-15T21:45:14.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:14.819 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:45:15.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892351019 2024-09-15T21:45:15.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892351019 2024-09-15T21:45:15.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836612 1-42949673085 2-64424509560 3-347892351019' 2024-09-15T21:45:15.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:15.212 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:45:15.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182502 2024-09-15T21:45:15.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182502 2024-09-15T21:45:15.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836612 1-42949673085 2-64424509560 3-347892351019 4-107374182502' 2024-09-15T21:45:15.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:15.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:45:15.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122086 2024-09-15T21:45:15.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122086 2024-09-15T21:45:15.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836612 1-42949673085 2-64424509560 3-347892351019 4-107374182502 5-377957122086' 2024-09-15T21:45:15.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:15.839 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:45:16.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533407 2024-09-15T21:45:16.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533407 2024-09-15T21:45:16.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836612 1-42949673085 2-64424509560 3-347892351019 4-107374182502 5-377957122086 6-455266533407' 2024-09-15T21:45:16.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:16.107 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:45:16.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691920 2024-09-15T21:45:16.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691920 2024-09-15T21:45:16.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836612 1-42949673085 2-64424509560 3-347892351019 4-107374182502 5-377957122086 6-455266533407 7-171798691920' 2024-09-15T21:45:16.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:16.506 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:45:16.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528396 2024-09-15T21:45:16.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528396 2024-09-15T21:45:16.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836612 1-42949673085 2-64424509560 3-347892351019 4-107374182502 5-377957122086 6-455266533407 7-171798691920 8-193273528396' 2024-09-15T21:45:16.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:16.664 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:45:16.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=485331304474 2024-09-15T21:45:16.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 485331304474 2024-09-15T21:45:16.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836612 1-42949673085 2-64424509560 3-347892351019 4-107374182502 5-377957122086 6-455266533407 7-171798691920 8-193273528396 9-485331304474' 2024-09-15T21:45:16.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:16.816 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T21:45:16.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201342 2024-09-15T21:45:16.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201342 2024-09-15T21:45:16.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836612 1-42949673085 2-64424509560 3-347892351019 4-107374182502 5-377957122086 6-455266533407 7-171798691920 8-193273528396 9-485331304474 10-236223201342' 2024-09-15T21:45:16.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:16.974 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836612 2024-09-15T21:45:16.974 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:16.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:45:16.976 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836612 2024-09-15T21:45:16.976 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:16.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836612 2024-09-15T21:45:16.977 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 21474836612 2024-09-15T21:45:16.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836612' 2024-09-15T21:45:16.977 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:45:17.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836613 -lt 21474836612 2024-09-15T21:45:17.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:17.403 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673085 2024-09-15T21:45:17.404 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:17.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:45:17.407 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673085 2024-09-15T21:45:17.407 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:17.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673085 2024-09-15T21:45:17.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673085' 2024-09-15T21:45:17.408 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 42949673085 2024-09-15T21:45:17.408 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:45:17.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673085 -lt 42949673085 2024-09-15T21:45:17.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:17.860 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509560 2024-09-15T21:45:17.860 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:17.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:45:17.862 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509560 2024-09-15T21:45:17.862 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:17.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509560 2024-09-15T21:45:17.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509560' 2024-09-15T21:45:17.864 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 64424509560 2024-09-15T21:45:17.864 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:45:18.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509561 -lt 64424509560 2024-09-15T21:45:18.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:18.278 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892351019 2024-09-15T21:45:18.278 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:18.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:45:18.280 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892351019 2024-09-15T21:45:18.280 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:18.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892351019 2024-09-15T21:45:18.281 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 347892351019 2024-09-15T21:45:18.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892351019' 2024-09-15T21:45:18.282 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:45:18.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892351020 -lt 347892351019 2024-09-15T21:45:18.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:18.875 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182502 2024-09-15T21:45:18.875 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:18.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:45:18.877 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182502 2024-09-15T21:45:18.877 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:18.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182502 2024-09-15T21:45:18.879 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 107374182502 2024-09-15T21:45:18.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182502' 2024-09-15T21:45:18.880 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:45:19.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182503 -lt 107374182502 2024-09-15T21:45:19.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:19.344 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122086 2024-09-15T21:45:19.345 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:19.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:45:19.347 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122086 2024-09-15T21:45:19.348 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:19.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122086 2024-09-15T21:45:19.349 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 377957122086 2024-09-15T21:45:19.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122086' 2024-09-15T21:45:19.349 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:45:19.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122087 -lt 377957122086 2024-09-15T21:45:19.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:19.797 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-455266533407 2024-09-15T21:45:19.797 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:19.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:45:19.800 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-455266533407 2024-09-15T21:45:19.800 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:19.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533407 2024-09-15T21:45:19.801 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 455266533407 2024-09-15T21:45:19.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 455266533407' 2024-09-15T21:45:19.802 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:45:20.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533408 -lt 455266533407 2024-09-15T21:45:20.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:20.242 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691920 2024-09-15T21:45:20.243 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:20.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:45:20.245 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691920 2024-09-15T21:45:20.245 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:20.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691920 2024-09-15T21:45:20.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691920' 2024-09-15T21:45:20.247 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.7 seq 171798691920 2024-09-15T21:45:20.247 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:45:20.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691920 -lt 171798691920 2024-09-15T21:45:20.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:20.675 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528396 2024-09-15T21:45:20.675 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:20.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:45:20.677 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528396 2024-09-15T21:45:20.678 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:20.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528396 2024-09-15T21:45:20.679 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.8 seq 193273528396 2024-09-15T21:45:20.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528396' 2024-09-15T21:45:20.679 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:45:21.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528397 -lt 193273528396 2024-09-15T21:45:21.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:21.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:21.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-485331304474 2024-09-15T21:45:21.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:45:21.116 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-485331304474 2024-09-15T21:45:21.116 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:21.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=485331304474 2024-09-15T21:45:21.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 485331304474' 2024-09-15T21:45:21.118 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.9 seq 485331304474 2024-09-15T21:45:21.119 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:45:21.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 485331304475 -lt 485331304474 2024-09-15T21:45:21.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:21.621 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201342 2024-09-15T21:45:21.622 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:21.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-15T21:45:21.625 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201342 2024-09-15T21:45:21.625 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:21.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201342 2024-09-15T21:45:21.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201342' 2024-09-15T21:45:21.627 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.10 seq 236223201342 2024-09-15T21:45:21.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T21:45:22.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201343 -lt 236223201342 2024-09-15T21:45:22.077 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:45:22.078 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:45:22.078 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:45:22.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-15T21:45:22.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:45:22.644 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:45:22.644 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:45:22.644 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:45:22.644 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:45:22.644 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:45:22.644 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:45:23.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=27 2024-09-15T21:45:23.046 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:45:23.047 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:45:23.047 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:45:23.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 27 = 28 2024-09-15T21:45:23.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 27 '!=' -1 2024-09-15T21:45:23.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:45:23.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=27 2024-09-15T21:45:23.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:45:23.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:45:23.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:45:23.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:45:23.767 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:45:23.767 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:45:23.767 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:45:23.767 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:45:23.768 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:45:23.768 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:45:24.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-15T21:45:24.177 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:45:24.178 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:45:24.178 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:45:24.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-15T21:45:24.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:45:24.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:45:24.741 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-15T21:45:24.741 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T21:45:24.741 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:45:24.741 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T21:45:24.741 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:45:25.196 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-15T21:45:25.196 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:45:25.196 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:45:25.196 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:45:25.196 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:45:25.197 INFO:tasks.workunit.client.0.smithi104.stderr:10' 2024-09-15T21:45:25.197 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2024-09-15T21:45:25.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2024-09-15T21:45:25.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2024-09-15T21:45:25.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:229: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2024-09-15T21:45:25.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-jerasure 2024-09-15T21:45:25.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T21:45:25.750 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T21:45:25.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2024-09-15T21:45:26.805 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile profile-jerasure does not exist 2024-09-15T21:45:26.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T21:45:26.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2024-09-15T21:45:26.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2024-09-15T21:45:26.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:157: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2024-09-15T21:45:26.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:158: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2024-09-15T21:45:26.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:160: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2024-09-15T21:45:27.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:165: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-15T21:45:27.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-15T21:45:29.630 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-lrc-a' already exists 2024-09-15T21:45:29.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:45:30.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:168: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2024-09-15T21:45:30.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T21:45:30.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2024-09-15T21:45:30.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T21:45:30.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:45:30.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T21:45:30.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:45:30.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T21:45:30.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:45:30.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T21:45:30.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:45:30.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T21:45:30.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T21:45:31.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:45:31.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:45:31.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T21:45:31.250 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-15T21:45:31.250 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc-a 2024-09-15T21:45:31.251 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:45:31.251 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-15T21:45:31.251 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:45:31.832 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-15T21:45:31.832 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:45:31.832 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T21:45:31.832 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 9 3 2024-09-15T21:45:31.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '9' '3') 2024-09-15T21:45:31.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T21:45:31.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-15T21:45:31.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2024-09-15T21:45:32.637 INFO:tasks.workunit.client.0.smithi104.stderr:osd.3 is already out. 2024-09-15T21:45:32.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-15T21:45:37.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-15T21:45:37.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc-a 2024-09-15T21:45:37.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:45:37.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<3\>' 2024-09-15T21:45:37.661 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:45:37.661 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-15T21:45:38.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-15T21:45:38.143 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:45:38.143 INFO:tasks.workunit.client.0.smithi104.stderr:7' 2024-09-15T21:45:38.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 9 7 2024-09-15T21:45:38.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:45:38.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:45:38.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 3 2024-09-15T21:45:38.716 INFO:tasks.workunit.client.0.smithi104.stderr:osd.3 is already in. 2024-09-15T21:45:38.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T21:45:38.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:170: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2024-09-15T21:45:38.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-lrc-a 2024-09-15T21:45:38.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2024-09-15T21:45:40.392 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-lrc-a' does not exist 2024-09-15T21:45:40.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2024-09-15T21:45:41.971 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile profile-lrc-a does not exist 2024-09-15T21:45:41.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T21:45:41.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2024-09-15T21:45:41.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2024-09-15T21:45:41.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:176: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2024-09-15T21:45:41.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:177: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2024-09-15T21:45:41.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2024-09-15T21:45:42.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:183: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2024-09-15T21:45:42.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2024-09-15T21:45:43.536 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-lrc' already exists 2024-09-15T21:45:43.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:45:44.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:186: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T21:45:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T21:45:46.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:45:46.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:45:46.742 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T21:45:46.745 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-15T21:45:46.745 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc 2024-09-15T21:45:46.745 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:45:46.745 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-15T21:45:46.745 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:45:47.361 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=8 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:10 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:7' 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 8 6 0 10 5 3 9 7 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('8' '6' '0' '10' '5' '3' '9' '7') 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2024-09-15T21:45:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2024-09-15T21:45:47.948 INFO:tasks.workunit.client.0.smithi104.stderr:osd.7 is already out. 2024-09-15T21:45:47.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-15T21:45:52.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-15T21:45:52.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc 2024-09-15T21:45:52.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:45:52.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2024-09-15T21:45:52.970 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-15T21:45:52.970 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:45:53.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=8 2024-09-15T21:45:53.445 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T21:45:53.445 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:45:53.445 INFO:tasks.workunit.client.0.smithi104.stderr:10 2024-09-15T21:45:53.445 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:45:53.445 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:45:53.445 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:45:53.445 INFO:tasks.workunit.client.0.smithi104.stderr:4' 2024-09-15T21:45:53.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 8 6 0 10 5 3 9 4 2024-09-15T21:45:53.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:45:53.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:45:53.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2024-09-15T21:45:54.048 INFO:tasks.workunit.client.0.smithi104.stderr:osd.7 is already in. 2024-09-15T21:45:54.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T21:45:54.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:188: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2024-09-15T21:45:54.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-lrc 2024-09-15T21:45:54.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2024-09-15T21:45:55.662 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-lrc' does not exist 2024-09-15T21:45:55.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2024-09-15T21:45:57.416 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile profile-lrc does not exist 2024-09-15T21:45:57.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T21:45:57.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2024-09-15T21:45:57.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:234: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2024-09-15T21:45:57.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:236: TEST_rados_put_get_shec: local poolname=pool-shec 2024-09-15T21:45:57.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:237: TEST_rados_put_get_shec: local profile=profile-shec 2024-09-15T21:45:57.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2024-09-15T21:45:58.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:243: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2024-09-15T21:45:58.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2024-09-15T21:45:59.512 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-shec' already exists 2024-09-15T21:45:59.531 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:46:00.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:246: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T21:46:00.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T21:46:01.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:46:02.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:46:02.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T21:46:02.192 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2024-09-15T21:46:02.192 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-shec 2024-09-15T21:46:02.192 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:46:02.193 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-15T21:46:02.193 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:46:02.655 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=7 2024-09-15T21:46:02.655 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:46:02.655 INFO:tasks.workunit.client.0.smithi104.stderr:1' 2024-09-15T21:46:02.655 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 7 9 1 2024-09-15T21:46:02.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('7' '9' '1') 2024-09-15T21:46:02.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T21:46:02.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-15T21:46:02.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2024-09-15T21:46:04.022 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 is already out. 2024-09-15T21:46:04.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-15T21:46:09.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-shec SOMETHING 2024-09-15T21:46:09.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<1\>' 2024-09-15T21:46:09.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-shec 2024-09-15T21:46:09.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:46:09.048 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-15T21:46:09.048 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:46:09.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=7 2024-09-15T21:46:09.519 INFO:tasks.workunit.client.0.smithi104.stderr:9 2024-09-15T21:46:09.519 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T21:46:09.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 7 9 6 2024-09-15T21:46:09.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-15T21:46:09.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T21:46:09.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 1 2024-09-15T21:46:10.126 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 is already in. 2024-09-15T21:46:10.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T21:46:10.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:248: TEST_rados_put_get_shec: delete_pool pool-shec 2024-09-15T21:46:10.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-shec 2024-09-15T21:46:10.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2024-09-15T21:46:12.150 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-shec' does not exist 2024-09-15T21:46:12.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2024-09-15T21:46:13.498 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile profile-shec does not exist 2024-09-15T21:46:13.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2024-09-15T21:46:13.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=ecpool 2024-09-15T21:46:13.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2024-09-15T21:46:14.269 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'ecpool' does not exist 2024-09-15T21:46:14.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2024-09-15T21:46:14.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2024-09-15T21:46:14.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:46:14.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2024-09-15T21:46:14.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:46:14.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:46:14.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:46:14.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:46:14.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:46:14.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:46:14.405 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:46:14.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:46:14.405 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:46:14.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:46:14.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:46:14.406 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:46:14.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:46:14.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:46:14.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:46:14.407 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:46:14.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:46:14.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:46:14.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2024-09-15T21:46:14.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:46:14.511 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:14.511 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:46:14.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.50061 2024-09-15T21:46:14.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:46:14.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:46:14.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T21:46:14.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/test-erasure-code 0 2024-09-15T21:46:14.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2024-09-15T21:46:14.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T21:46:14.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2024-09-15T21:46:14.513 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:46:14.513 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:46:14.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:46:14.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:46:14.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:46:14.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:46:14.514 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:46:14.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:46:14.515 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:46:14.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:46:14.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:46:14.516 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:46:14.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:46:14.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:46:14.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:46:14.517 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:46:14.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:46:14.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T21:46:14.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2024-09-15T21:46:14.520 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:46:14.520 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:14.520 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50061 2024-09-15T21:46:14.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.50061 2024-09-15T21:46:14.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:46:14.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:46:14.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T21:46:14.523 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T21:46:14.523 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T21:46:14.534 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2024-09-15T21:46:14.534 DEBUG:teuthology.orchestra.run.smithi104:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2024-09-15T21:46:14.588 INFO:tasks.workunit.client.0.smithi104.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T21:46:14.594 INFO:tasks.workunit.client.0.smithi104.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T21:46:14.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T21:46:14.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T21:46:14.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T21:46:14.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T21:46:14.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T21:46:14.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T21:46:14.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T21:46:14.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T21:46:14.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/test-erasure-eio 2024-09-15T21:46:14.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2024-09-15T21:46:14.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2024-09-15T21:46:14.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2024-09-15T21:46:14.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2024-09-15T21:46:14.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2024-09-15T21:46:14.595 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2024-09-15T21:46:14.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none ' 2024-09-15T21:46:14.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2024-09-15T21:46:14.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-09-15T21:46:14.596 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2024-09-15T21:46:14.596 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T21:46:14.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2024-09-15T21:46:14.598 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_ec_object_attr_read_error 2024-09-15T21:46:14.598 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_ec_recovery_multiple_errors 2024-09-15T21:46:14.598 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_ec_recovery_multiple_objects 2024-09-15T21:46:14.598 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_ec_recovery_multiple_objects_eio 2024-09-15T21:46:14.598 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_ec_recovery_unfound 2024-09-15T21:46:14.599 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_ec_single_recovery_error 2024-09-15T21:46:14.599 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_get_bad_size_shard_0 2024-09-15T21:46:14.599 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_get_bad_size_shard_1 2024-09-15T21:46:14.599 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_get_subread_eio_shard_0 2024-09-15T21:46:14.599 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_get_subread_eio_shard_1 2024-09-15T21:46:14.599 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_get_subread_missing 2024-09-15T21:46:14.599 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2024-09-15T21:46:14.599 INFO:tasks.workunit.client.0.smithi104.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2024-09-15T21:46:14.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T21:46:14.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T21:46:14.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T21:46:14.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T21:46:14.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T21:46:14.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:46:14.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T21:46:14.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:46:14.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:46:14.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:46:14.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:46:14.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:46:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:46:14.601 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:46:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:46:14.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:46:14.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:46:14.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:46:14.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:46:14.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:46:14.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:46:14.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:46:14.603 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:46:14.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:46:14.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:46:14.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T21:46:14.610 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:46:14.610 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:14.610 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:46:14.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T21:46:14.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:46:14.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:46:14.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T21:46:14.611 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:46:14.611 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:14.611 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:46:14.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T21:46:14.611 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:46:14.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T21:46:14.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T21:46:14.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:46:14.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T21:46:14.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T21:46:14.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T21:46:14.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:46:14.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:46:14.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:46:14.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T21:46:14.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T21:46:15.026 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:46:15.026 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:46:15.026 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:46:15.026 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:46:15.026 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:15.027 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:46:15.027 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:46:15.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:46:15.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:46:15.236 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:46:15.236 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:46:15.236 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:46:15.237 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:46:15.237 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:46:15.237 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:46:15.238 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:46:15.238 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:46:15.238 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:46:15.238 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:15.238 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:46:15.238 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T21:46:15.238 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:46:15.238 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T21:46:15.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:46:15.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:46:15.353 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:46:15.353 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:46:15.354 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:46:15.354 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:46:15.354 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:46:15.354 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:46:15.354 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:46:15.354 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:15.354 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:46:15.354 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T21:46:15.355 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:46:15.355 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T21:46:15.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T21:46:15.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T21:46:15.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:46:15.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:46:15.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:46:15.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T21:46:15.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:46:16.356 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:46:16.357 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:46:16.357 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:46:16.357 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:46:16.357 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:16.357 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:46:16.358 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:46:16.358 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:46:16.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:46:16.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T21:46:16.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:46:16.705 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T21:46:16.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:46:17.723 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T21:46:17.723 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:46:17.723 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:46:17.724 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:46:17.724 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:17.724 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:46:17.724 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T21:46:17.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T21:46:17.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T21:46:17.823 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:46:17.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T21:46:17.837 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:46:15.066+0000 7fd2325b4e00 0 load: jerasure load: lrc 2024-09-15T21:46:17.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2024-09-15T21:46:17.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2024-09-15T21:46:17.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: local objname=myobject 2024-09-15T21:46:17.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: local lastobj=300 2024-09-15T21:46:17.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:535: TEST_ec_backfill_unfound: local testobj=obj250 2024-09-15T21:46:17.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:46:17.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T21:46:17.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: setup_osds 5 2024-09-15T21:46:17.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-15T21:46:17.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T21:46:17.839 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-15T21:46:17.840 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-15T21:46:17.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:46:17.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T21:46:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:46:17.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e62fbe00-2d57-4a33-a381-30a5022228bb 2024-09-15T21:46:17.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e62fbe00-2d57-4a33-a381-30a5022228bb' 2024-09-15T21:46:17.848 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 e62fbe00-2d57-4a33-a381-30a5022228bb 2024-09-15T21:46:17.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:46:17.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCpVedmV7GAMxAAWicDEcJ10QNP8CuVInsf/Q== 2024-09-15T21:46:17.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCpVedmV7GAMxAAWicDEcJ10QNP8CuVInsf/Q=="}' 2024-09-15T21:46:17.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e62fbe00-2d57-4a33-a381-30a5022228bb -i td/test-erasure-eio/0/new.json 2024-09-15T21:46:18.164 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:46:18.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T21:46:18.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpVedmV7GAMxAAWicDEcJ10QNP8CuVInsf/Q== --osd-uuid e62fbe00-2d57-4a33-a381-30a5022228bb 2024-09-15T21:46:18.213 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:18.210+0000 7f7796e297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:18.224 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:18.222+0000 7f7796e297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:18.225 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:18.222+0000 7f7796e297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:18.225 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:18.222+0000 7f7796e297c0 -1 bdev(0x564f99949c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:46:18.225 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:18.222+0000 7f7796e297c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T21:46:23.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T21:46:23.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:46:23.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:46:23.679 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:46:23.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:46:24.284 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:46:24.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:46:24.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:46:24.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:46:24.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:46:24.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:46:24.321 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:24.318+0000 7faf9e8a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:24.363 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:24.358+0000 7faf9e8a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:24.364 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:24.362+0000 7faf9e8a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:24.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:46:24.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:46:24.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:46:24.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:46:24.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:46:24.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:24.738 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:46:24.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:46:24.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:24.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:25.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:25.846 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:25.842+0000 7faf9e8a07c0 -1 Falling back to public interface 2024-09-15T21:46:26.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:26.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:26.183 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:46:26.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:46:26.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:26.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:26.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:27.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:27.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:27.617 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:46:27.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:46:27.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:27.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:28.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:28.584 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:28.582+0000 7faf9e8a07c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:46:29.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:29.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:29.035 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:46:29.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:46:29.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:29.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:29.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:30.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:30.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:30.570 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:46:30.571 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:46:30.571 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:30.571 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:30.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:32.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:32.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:32.001 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:46:32.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:46:32.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:32.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:32.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:33.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:33.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:33.484 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:46:33.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:46:33.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:33.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:33.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:34.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:34.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:34.929 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:46:34.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:46:34.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:34.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:35.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:36.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:36.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:36.412 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:46:36.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:46:36.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:36.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:36.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:37.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:37.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:37.878 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:46:37.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:46:37.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:37.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:38.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:39.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:39.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:39.449 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:46:39.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:46:39.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:39.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:39.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:40.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:40.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:40.878 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:46:40.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:46:40.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:40.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:41.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:42.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:42.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:42.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:46:42.393 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:46:42.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:42.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:43.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:44.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:44.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:44.109 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:46:44.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:46:44.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:44.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:44.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:45.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:45.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:45.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:46:45.873 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:46:45.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:45.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:46.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:47.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:47.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:47.791 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:46:47.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:46:47.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:47.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:48.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:49.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:49.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:49.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:46:49.350 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:46:49.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:49.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:49.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:50.935 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:50.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:50.936 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:46:50.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:46:50.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:50.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:51.397 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1365352289,v1:127.0.0.1:6803/1365352289] [v2:127.0.0.1:6804/1365352289,v1:127.0.0.1:6805/1365352289] exists,up e62fbe00-2d57-4a33-a381-30a5022228bb 2024-09-15T21:46:51.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:46:51.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:46:51.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:46:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:46:51.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:46:51.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:46:51.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:46:51.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:46:51.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:46:51.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:46:51.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:46:51.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:46:51.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:46:51.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:46:51.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:46:51.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T21:46:51.404 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:46:51.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=388aceff-7753-420b-8a06-3f4047251e87 2024-09-15T21:46:51.404 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 388aceff-7753-420b-8a06-3f4047251e87 2024-09-15T21:46:51.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 388aceff-7753-420b-8a06-3f4047251e87' 2024-09-15T21:46:51.405 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:46:51.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDLVedmpP8nGRAAp/Ae6emCFHgjI+S22eQg8Q== 2024-09-15T21:46:51.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDLVedmpP8nGRAAp/Ae6emCFHgjI+S22eQg8Q=="}' 2024-09-15T21:46:51.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 388aceff-7753-420b-8a06-3f4047251e87 -i td/test-erasure-eio/1/new.json 2024-09-15T21:46:51.960 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:46:51.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T21:46:51.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLVedmpP8nGRAAp/Ae6emCFHgjI+S22eQg8Q== --osd-uuid 388aceff-7753-420b-8a06-3f4047251e87 2024-09-15T21:46:52.015 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:52.010+0000 7fbbfb3ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:52.027 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:52.022+0000 7fbbfb3ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:52.028 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:52.026+0000 7fbbfb3ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:52.029 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:52.026+0000 7fbbfb3ff7c0 -1 bdev(0x555e8fc1bc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:46:52.029 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:52.026+0000 7fbbfb3ff7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T21:46:58.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T21:46:58.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:46:58.225 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T21:46:58.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:46:58.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:46:58.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:46:58.809 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T21:46:58.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:46:58.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:46:58.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:46:58.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:46:58.838 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:58.834+0000 7f3d8a8027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:58.840 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:58.838+0000 7f3d8a8027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:58.841 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:46:58.838+0000 7f3d8a8027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:59.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:46:59.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:46:59.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:46:59.231 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:46:59.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:46:59.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:46:59.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:59.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:46:59.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:59.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:46:59.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:00.094 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:00.090+0000 7f3d8a8027c0 -1 Falling back to public interface 2024-09-15T21:47:00.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:00.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:00.714 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:47:00.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:47:00.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:00.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:01.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:02.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:02.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:02.229 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:47:02.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:47:02.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:02.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:02.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:02.903 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:02.898+0000 7f3d8a8027c0 -1 osd.1 0 log_to_monitors true 2024-09-15T21:47:03.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:03.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:03.678 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:47:03.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:47:03.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:03.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:04.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:05.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:05.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:05.126 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:47:05.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:47:05.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:05.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:05.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:06.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:06.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:06.700 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:47:06.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:47:06.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:06.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:07.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:08.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:08.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:08.190 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:47:08.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:47:08.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:08.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:08.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:09.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:09.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:09.636 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:47:09.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:47:09.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:09.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:10.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:11.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:11.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:11.115 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:47:11.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:47:11.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:11.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:11.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:12.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:12.561 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:47:12.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:12.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:47:12.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:12.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:13.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:14.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:14.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:14.065 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:47:14.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:47:14.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:14.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:14.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:15.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:15.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:15.500 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:47:15.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:47:15.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:15.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:16.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:17.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:17.309 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:47:17.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:17.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:47:17.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:17.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:17.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:18.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:18.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:18.734 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:47:18.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:47:18.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:18.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:19.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:20.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:20.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:20.423 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:47:20.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:47:20.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:20.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:21.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:22.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:22.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:22.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:47:22.153 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:47:22.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:22.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:23.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:24.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:24.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:24.022 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:47:24.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:47:24.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:24.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:25.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:26.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:26.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:26.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:47:26.072 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:47:26.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:26.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:47:26.621 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1032271160,v1:127.0.0.1:6811/1032271160] [v2:127.0.0.1:6812/1032271160,v1:127.0.0.1:6813/1032271160] exists,up 388aceff-7753-420b-8a06-3f4047251e87 2024-09-15T21:47:26.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:47:26.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:47:26.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:47:26.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:47:26.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:47:26.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:47:26.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:47:26.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:47:26.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:47:26.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:47:26.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:47:26.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:47:26.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:47:26.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T21:47:26.629 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:47:26.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8a6b43e9-5157-4016-a80c-69cfcd431874 2024-09-15T21:47:26.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8a6b43e9-5157-4016-a80c-69cfcd431874' 2024-09-15T21:47:26.629 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 8a6b43e9-5157-4016-a80c-69cfcd431874 2024-09-15T21:47:26.629 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:47:26.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDuVedmKQ73JhAA2//I5KVBWiNdIvvtPxBepA== 2024-09-15T21:47:26.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDuVedmKQ73JhAA2//I5KVBWiNdIvvtPxBepA=="}' 2024-09-15T21:47:26.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8a6b43e9-5157-4016-a80c-69cfcd431874 -i td/test-erasure-eio/2/new.json 2024-09-15T21:47:27.392 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:47:27.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T21:47:27.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuVedmKQ73JhAA2//I5KVBWiNdIvvtPxBepA== --osd-uuid 8a6b43e9-5157-4016-a80c-69cfcd431874 2024-09-15T21:47:27.452 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:27.450+0000 7f0a766097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:47:27.455 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:27.450+0000 7f0a766097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:47:27.456 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:27.454+0000 7f0a766097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:47:27.456 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:27.454+0000 7f0a766097c0 -1 bdev(0x55ff55c77c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:47:27.456 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:27.454+0000 7f0a766097c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T21:47:33.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T21:47:33.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:47:33.051 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T21:47:33.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:47:33.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:47:33.633 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T21:47:33.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:47:33.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:47:33.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:47:33.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:47:33.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:47:33.669 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:33.666+0000 7f31e712c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:47:33.695 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:33.690+0000 7f31e712c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:47:33.696 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:33.694+0000 7f31e712c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:47:34.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:47:34.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:47:34.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:47:34.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:47:34.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:47:34.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:34.133 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:47:34.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:47:34.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:34.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:34.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:34.730 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:34.726+0000 7f31e712c7c0 -1 Falling back to public interface 2024-09-15T21:47:35.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:35.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:35.610 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:47:35.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:47:35.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:35.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:37.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:37.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:37.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:47:37.107 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:47:37.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:37.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:37.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:37.702 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:47:37.698+0000 7f31e712c7c0 -1 osd.2 0 log_to_monitors true 2024-09-15T21:47:38.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:38.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:38.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:47:38.563 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:47:38.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:38.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:39.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:40.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:40.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:40.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:47:40.043 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:47:40.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:40.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:40.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:41.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:41.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:41.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:47:41.511 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:47:41.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:41.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:41.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:42.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:42.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:42.964 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:47:42.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:47:42.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:42.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:43.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:44.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:44.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:44.521 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:47:44.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:47:44.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:44.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:44.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:45.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:45.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:45.957 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:47:45.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:47:45.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:45.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:46.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:47.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:47.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:47.507 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:47:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:47:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:47.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:48.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:48.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:48.941 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:47:48.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:47:48.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:48.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:49.490 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:50.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:50.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:50.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:47:50.493 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:47:50.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:50.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:51.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:52.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:52.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:52.461 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:47:52.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:47:52.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:52.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:53.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:54.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:54.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:54.138 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:47:54.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:47:54.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:54.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:54.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:55.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:55.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:55.963 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:47:55.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:47:55.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:55.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:57.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:58.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:58.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:58.004 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:47:58.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:47:58.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:58.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:47:58.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:59.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:59.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:59.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:47:59.952 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:47:59.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:59.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:48:00.401 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3112912778,v1:127.0.0.1:6819/3112912778] [v2:127.0.0.1:6820/3112912778,v1:127.0.0.1:6821/3112912778] exists,up 8a6b43e9-5157-4016-a80c-69cfcd431874 2024-09-15T21:48:00.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:48:00.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:48:00.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:48:00.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:48:00.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T21:48:00.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:48:00.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:48:00.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T21:48:00.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:48:00.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T21:48:00.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T21:48:00.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:48:00.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:48:00.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:48:00.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T21:48:00.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T21:48:00.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:48:00.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:48:00.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:48:00.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:48:00.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:48:00.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:48:00.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:48:00.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:48:00.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:48:00.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T21:48:00.408 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:48:00.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7c91e332-bfba-47cf-961a-22af59f3e6ce 2024-09-15T21:48:00.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 7c91e332-bfba-47cf-961a-22af59f3e6ce' 2024-09-15T21:48:00.408 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 7c91e332-bfba-47cf-961a-22af59f3e6ce 2024-09-15T21:48:00.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:48:00.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAQVudmCcS4GRAAy0ATtzkDr9fziUdl1fda4w== 2024-09-15T21:48:00.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAQVudmCcS4GRAAy0ATtzkDr9fziUdl1fda4w=="}' 2024-09-15T21:48:00.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7c91e332-bfba-47cf-961a-22af59f3e6ce -i td/test-erasure-eio/3/new.json 2024-09-15T21:48:01.397 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:48:01.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T21:48:01.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQVudmCcS4GRAAy0ATtzkDr9fziUdl1fda4w== --osd-uuid 7c91e332-bfba-47cf-961a-22af59f3e6ce 2024-09-15T21:48:01.453 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:01.449+0000 7fdf0ea2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:01.470 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:01.465+0000 7fdf0ea2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:01.471 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:01.465+0000 7fdf0ea2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:01.471 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:01.469+0000 7fdf0ea2a7c0 -1 bdev(0x55d12556bc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T21:48:01.471 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:01.469+0000 7fdf0ea2a7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T21:48:07.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T21:48:07.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:48:07.918 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T21:48:07.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T21:48:07.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:48:08.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T21:48:08.515 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T21:48:08.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:48:08.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:48:08.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:48:08.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:48:08.551 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:08.545+0000 7f0feba9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:08.556 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:08.549+0000 7f0feba9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:08.557 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:08.553+0000 7f0feba9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:08.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T21:48:08.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:48:08.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:48:08.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:48:08.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:48:08.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:08.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:48:08.922 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:48:08.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:08.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:09.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:09.882 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:09.877+0000 7f0feba9e7c0 -1 Falling back to public interface 2024-09-15T21:48:10.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:10.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:10.377 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:48:10.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:48:10.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:10.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:10.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:11.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:11.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:11.837 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:48:11.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:48:11.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:11.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:12.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:12.748 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:12.745+0000 7f0feba9e7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T21:48:13.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:13.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:13.292 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:48:13.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:48:13.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:13.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:13.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:14.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:14.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:14.932 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:48:14.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:48:14.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:14.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:15.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:16.571 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:16.571 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:16.571 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:48:16.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:48:16.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:16.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:17.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:18.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:18.032 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:48:18.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:18.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:48:18.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:18.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:18.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:19.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:19.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:19.657 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:48:19.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:48:19.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:19.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:20.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:21.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:21.104 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:48:21.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:21.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:48:21.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:21.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:21.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:22.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:22.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:22.671 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:48:22.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:48:22.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:22.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:23.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:24.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:24.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:24.106 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:48:24.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:48:24.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:24.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:24.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:25.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:25.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:25.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:48:25.638 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:48:25.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:25.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:26.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:27.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:27.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:27.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:48:27.238 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:48:27.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:27.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:28.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:29.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:29.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:29.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:48:29.410 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:48:29.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:29.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:29.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:30.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:30.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:30.940 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:48:30.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:48:30.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:30.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:31.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:32.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:32.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:32.556 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:48:32.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:48:32.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:32.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:33.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:34.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:34.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:34.192 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:48:34.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:48:34.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:34.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:34.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:35.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:35.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:35.766 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:48:35.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:48:35.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:35.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:37.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:37.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:37.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:48:37.251 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:48:37.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:37.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:48:37.771 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3484974818,v1:127.0.0.1:6827/3484974818] [v2:127.0.0.1:6828/3484974818,v1:127.0.0.1:6829/3484974818] exists,up 7c91e332-bfba-47cf-961a-22af59f3e6ce 2024-09-15T21:48:37.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:48:37.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:48:37.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:48:37.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:48:37.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:48:37.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:48:37.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:48:37.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:48:37.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:48:37.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:48:37.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:48:37.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:48:37.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:48:37.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:48:37.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T21:48:37.781 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:48:37.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=782b0986-5bf9-4343-b4ab-fdb9f0b3fb25 2024-09-15T21:48:37.781 INFO:tasks.workunit.client.0.smithi104.stdout:add osd4 782b0986-5bf9-4343-b4ab-fdb9f0b3fb25 2024-09-15T21:48:37.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 782b0986-5bf9-4343-b4ab-fdb9f0b3fb25' 2024-09-15T21:48:37.781 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:48:37.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1VudmpzzgLxAA/9sMvTNlgyQxTbaJGwXswg== 2024-09-15T21:48:37.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1VudmpzzgLxAA/9sMvTNlgyQxTbaJGwXswg=="}' 2024-09-15T21:48:37.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 782b0986-5bf9-4343-b4ab-fdb9f0b3fb25 -i td/test-erasure-eio/4/new.json 2024-09-15T21:48:38.699 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:48:38.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T21:48:38.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1VudmpzzgLxAA/9sMvTNlgyQxTbaJGwXswg== --osd-uuid 782b0986-5bf9-4343-b4ab-fdb9f0b3fb25 2024-09-15T21:48:38.750 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:38.745+0000 7f26e19917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:38.778 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:38.773+0000 7f26e19917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:38.779 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:38.777+0000 7f26e19917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:38.779 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:38.777+0000 7f26e19917c0 -1 bdev(0x5597e042fc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-15T21:48:38.779 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:38.777+0000 7f26e19917c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T21:48:45.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T21:48:45.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:48:45.398 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd4 key to auth repository 2024-09-15T21:48:45.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-15T21:48:45.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:48:46.037 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.4 2024-09-15T21:48:46.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-15T21:48:46.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:48:46.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:48:46.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:48:46.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:48:46.073 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:46.069+0000 7f77ff9487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:46.075 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:46.073+0000 7f77ff9487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:46.076 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:46.073+0000 7f77ff9487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:46.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-15T21:48:46.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:48:46.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-15T21:48:46.509 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:48:46.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:48:46.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:48:46.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:46.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:48:46.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:46.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:48:47.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:47.110 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:47.105+0000 7f77ff9487c0 -1 Falling back to public interface 2024-09-15T21:48:48.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:48.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:48.022 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:48:48.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:48:48.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:48.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:48:48.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:49.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:49.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:49.500 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:48:49.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:48:49.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:49.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:48:49.761 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:48:49.757+0000 7f77ff9487c0 -1 osd.4 0 log_to_monitors true 2024-09-15T21:48:49.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:50.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:50.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:50.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:48:50.975 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:48:50.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:50.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:48:51.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:52.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:52.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:52.669 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:48:52.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:48:52.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:52.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:48:53.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:54.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:54.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:54.345 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:48:54.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:48:54.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:54.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:48:54.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:55.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:55.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:55.885 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:48:55.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:48:55.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:55.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:48:56.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:57.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:57.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:57.321 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:48:57.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:48:57.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:57.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:48:57.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:58.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:58.955 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:48:58.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:58.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:48:58.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:58.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:48:59.571 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:00.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:00.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:00.573 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:49:00.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:49:00.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:00.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:49:01.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:02.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:02.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:02.021 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:49:02.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:49:02.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:02.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:49:02.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:03.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:03.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:03.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:49:03.589 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:49:03.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:03.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:49:04.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:05.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:05.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:05.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:49:05.562 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:49:05.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:05.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:49:06.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:07.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:07.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:07.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:49:07.112 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:49:07.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:07.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:49:08.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:09.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:09.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:09.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:49:09.118 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:49:09.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:09.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:49:09.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:10.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:10.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:10.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:49:10.797 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:49:10.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:10.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:49:11.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:12.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:12.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:12.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:49:12.626 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:49:12.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:12.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:49:13.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:14.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:14.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:14.554 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:49:14.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:49:14.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:14.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:49:15.170 INFO:tasks.workunit.client.0.smithi104.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3338647599,v1:127.0.0.1:6835/3338647599] [v2:127.0.0.1:6836/3338647599,v1:127.0.0.1:6837/3338647599] exists,up 782b0986-5bf9-4343-b4ab-fdb9f0b3fb25 2024-09-15T21:49:15.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:49:15.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:49:15.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:49:15.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T21:49:15.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:49:15.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:49:15.172 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:49:15.172 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:49:15.172 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:49:15.173 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T21:49:15.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T21:49:15.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T21:49:15.281 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:49:15.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T21:49:15.297 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:46:25.854+0000 7faf9e8a07c0 0 load: jerasure load: lrc 2024-09-15T21:49:15.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:540: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:49:15.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:542: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2024-09-15T21:49:15.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:543: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T21:49:15.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T21:49:15.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T21:49:15.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T21:49:15.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T21:49:15.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T21:49:15.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T21:49:15.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T21:49:16.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T21:49:16.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T21:49:17.986 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T21:49:18.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:49:19.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T21:49:19.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:49:19.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:49:19.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:49:19.009 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:49:19.010 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:49:19.011 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:49:19.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:49:19.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:49:19.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:49:19.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:49:19.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:49:19.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:49:19.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:49:19.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:49:19.131 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:49:19.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:49:19.616 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:49:19.616 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:49:19.616 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:49:19.616 INFO:tasks.workunit.client.0.smithi104.stderr:4' 2024-09-15T21:49:19.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:49:19.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:19.617 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:49:19.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803811 2024-09-15T21:49:19.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803811 2024-09-15T21:49:19.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811' 2024-09-15T21:49:19.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:19.774 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:49:19.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574879 2024-09-15T21:49:19.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574879 2024-09-15T21:49:19.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574879' 2024-09-15T21:49:19.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:19.916 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:49:20.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345942 2024-09-15T21:49:20.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345942 2024-09-15T21:49:20.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574879 2-85899345942' 2024-09-15T21:49:20.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:20.065 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:49:20.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018899 2024-09-15T21:49:20.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018899 2024-09-15T21:49:20.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574879 2-85899345942 3-128849018899' 2024-09-15T21:49:20.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:20.212 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:49:20.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659146 2024-09-15T21:49:20.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659146 2024-09-15T21:49:20.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574879 2-85899345942 3-128849018899 4-176093659146' 2024-09-15T21:49:20.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:20.351 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803811 2024-09-15T21:49:20.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:20.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:49:20.354 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803811 2024-09-15T21:49:20.354 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:20.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803811 2024-09-15T21:49:20.355 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803811 2024-09-15T21:49:20.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-09-15T21:49:20.356 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:20.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803809 -lt 25769803811 2024-09-15T21:49:20.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:21.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:49:21.768 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:22.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803812 -lt 25769803811 2024-09-15T21:49:22.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:22.198 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:22.198 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574879 2024-09-15T21:49:22.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:49:22.200 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574879 2024-09-15T21:49:22.200 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:22.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574879 2024-09-15T21:49:22.201 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574879 2024-09-15T21:49:22.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574879' 2024-09-15T21:49:22.202 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:49:22.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574879 -lt 55834574879 2024-09-15T21:49:22.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:22.649 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345942 2024-09-15T21:49:22.649 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:22.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:49:22.652 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345942 2024-09-15T21:49:22.652 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:22.653 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 85899345942 2024-09-15T21:49:22.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345942 2024-09-15T21:49:22.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345942' 2024-09-15T21:49:22.657 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:49:23.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345943 -lt 85899345942 2024-09-15T21:49:23.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:23.141 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:23.141 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018899 2024-09-15T21:49:23.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:49:23.143 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018899 2024-09-15T21:49:23.143 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:23.145 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 128849018899 2024-09-15T21:49:23.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018899 2024-09-15T21:49:23.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018899' 2024-09-15T21:49:23.145 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:49:23.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018899 -lt 128849018899 2024-09-15T21:49:23.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:23.623 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-176093659146 2024-09-15T21:49:23.623 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:23.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:49:23.626 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:23.626 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-176093659146 2024-09-15T21:49:23.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659146 2024-09-15T21:49:23.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 176093659146' 2024-09-15T21:49:23.627 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 176093659146 2024-09-15T21:49:23.628 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:49:24.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659146 -lt 176093659146 2024-09-15T21:49:24.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:49:24.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:24.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:24.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:49:24.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:49:24.587 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:49:24.587 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:49:24.587 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:49:24.588 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:49:24.588 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:49:24.588 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:49:24.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:49:24.996 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:49:24.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:24.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:25.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:49:25.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:49:25.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:49:25.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-15T21:49:25.953 INFO:tasks.workunit.client.0.smithi104.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T21:49:25.953 INFO:tasks.workunit.client.0.smithi104.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:49:18.871509+0000 0'0 48:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-15T21:49:17.512572+0000 0'0 2024-09-15T21:49:17.512572+0000 0 0 periodic scrub scheduled @ 2024-09-16T22:32:14.170233+0000 0 0 2024-09-15T21:49:25.953 INFO:tasks.workunit.client.0.smithi104.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:49:17.535855+0000 0'0 47:18 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T01:46:12.152270+0000 0 0 2024-09-15T21:49:25.953 INFO:tasks.workunit.client.0.smithi104.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:49:17.007547+0000 0'0 47:112 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T08:05:50.396986+0000 0 0 2024-09-15T21:49:25.953 INFO:tasks.workunit.client.0.smithi104.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:49:17.191647+0000 0'0 47:18 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T06:23:23.885703+0000 0 0 2024-09-15T21:49:25.953 INFO:tasks.workunit.client.0.smithi104.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:48:02.070183+0000 0'0 48:75 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T05:05:21.875269+0000 0 0 2024-09-15T21:49:25.954 INFO:tasks.workunit.client.0.smithi104.stdout: 2024-09-15T21:49:25.954 INFO:tasks.workunit.client.0.smithi104.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T21:49:25.954 INFO:tasks.workunit.client.0.smithi104.stderr:dumped pgs 2024-09-15T21:49:25.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-15T21:49:25.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T21:49:25.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T21:49:25.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-15T21:49:25.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T21:49:25.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T21:49:25.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T21:49:25.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T21:49:25.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T21:49:25.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T21:49:25.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T21:49:25.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T21:49:25.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:26.157 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2024-09-15T21:49:26.157 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=pool-jerasure 2024-09-15T21:49:26.157 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=myobject 2024-09-15T21:49:26.157 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:49:26.157 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-15T21:49:26.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: local primary=3 2024-09-15T21:49:26.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2024-09-15T21:49:26.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T21:49:26.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-15T21:49:26.599 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T21:49:26.599 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:49:27.102 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T21:49:27.102 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:49:27.102 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:49:27.102 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:49:27.102 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T21:49:27.102 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2 2024-09-15T21:49:27.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-09-15T21:49:27.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: local -a initial_osds 2024-09-15T21:49:27.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: local last_osd=2 2024-09-15T21:49:27.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T21:49:27.103 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:49:27.103 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:49:27.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:49:27.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:49:27.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:49:27.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:49:27.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: ceph osd down 2 2024-09-15T21:49:28.307 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 is already down. 2024-09-15T21:49:28.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: ceph osd out 2 2024-09-15T21:49:28.979 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 is already out. 2024-09-15T21:49:28.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-15T21:49:29.422 INFO:tasks.workunit.client.0.smithi104.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T21:49:29.422 INFO:tasks.workunit.client.0.smithi104.stdout:2.0 1 0 1 0 0 4096 0 0 1 0 1 activating+undersized 2024-09-15T21:49:28.154831+0000 48'1 51:21 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-09-15T21:49:17.512572+0000 0'0 2024-09-15T21:49:17.512572+0000 0 0 no scrub is scheduled 0 0 2024-09-15T21:49:29.422 INFO:tasks.workunit.client.0.smithi104.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:49:17.535855+0000 0'0 48:20 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T01:46:12.152270+0000 0 0 2024-09-15T21:49:29.422 INFO:tasks.workunit.client.0.smithi104.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:49:17.007547+0000 0'0 48:114 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T08:05:50.396986+0000 0 0 2024-09-15T21:49:29.422 INFO:tasks.workunit.client.0.smithi104.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:49:17.191647+0000 0'0 48:20 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T06:23:23.885703+0000 0 0 2024-09-15T21:49:29.422 INFO:tasks.workunit.client.0.smithi104.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:48:02.070183+0000 0'0 48:76 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T05:05:21.875269+0000 0 0 2024-09-15T21:49:29.422 INFO:tasks.workunit.client.0.smithi104.stdout: 2024-09-15T21:49:29.422 INFO:tasks.workunit.client.0.smithi104.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T21:49:29.422 INFO:tasks.workunit.client.0.smithi104.stderr:dumped pgs 2024-09-15T21:49:29.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-15T21:49:29.439 INFO:tasks.workunit.client.0.smithi104.stderr:4+0 records in 2024-09-15T21:49:29.439 INFO:tasks.workunit.client.0.smithi104.stderr:4+0 records out 2024-09-15T21:49:29.439 INFO:tasks.workunit.client.0.smithi104.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00017392 s, 23.6 MB/s 2024-09-15T21:49:29.440 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: seq 1 300 2024-09-15T21:49:29.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:29.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:29.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:29.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:30.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:30.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:30.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:30.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:30.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:30.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:31.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:31.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:31.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:31.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:31.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:31.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:31.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:31.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:32.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:32.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:32.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:32.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:32.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:32.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:32.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:32.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:33.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:33.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:33.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:33.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:33.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:33.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:33.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:33.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:34.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:34.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:34.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:34.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:34.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:34.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:34.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:34.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:35.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:35.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:35.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:35.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:36.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:36.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:36.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:36.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:36.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:36.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:36.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:36.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:37.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:37.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:37.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:37.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:37.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:37.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:37.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:37.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:38.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:38.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:38.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:38.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:38.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:38.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:38.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:38.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:39.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:39.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:39.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:39.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:39.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:39.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:39.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:39.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:40.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:40.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:40.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:40.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:40.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:40.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:40.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:40.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:41.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:41.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:41.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:41.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:41.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:41.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:41.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:41.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:42.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:42.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:42.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:42.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:42.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:42.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:42.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:42.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:43.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:43.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:43.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:43.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:43.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:43.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:43.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:43.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:44.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:44.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:44.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:44.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:44.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:44.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:44.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:44.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:45.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:45.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:45.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:45.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:45.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:45.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:45.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:45.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:46.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:46.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:46.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:46.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:46.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:46.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:46.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:46.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:47.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:47.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:47.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:47.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:47.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:47.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:47.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:47.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:48.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:48.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:48.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:48.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:48.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:48.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:48.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:48.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:49.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:49.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:49.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:49.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:49.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:49.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:49.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:49.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:50.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:50.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:50.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:50.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:50.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:50.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:50.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:50.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:51.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:51.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:51.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:51.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:51.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:51.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:52.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:52.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:52.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:52.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:52.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:52.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:52.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:52.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:53.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:53.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:53.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:53.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:53.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:53.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:54.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:54.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:54.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:54.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:54.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:54.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:54.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:54.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:55.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:55.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:55.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:55.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:55.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:55.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:55.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:55.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:56.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:56.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:56.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:56.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:56.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:56.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:56.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:56.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:57.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:57.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:57.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:57.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:57.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:57.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:57.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:57.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:58.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:58.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:58.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:58.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:58.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:58.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:58.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:58.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:59.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:59.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:59.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:59.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:59.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:59.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2024-09-15T21:49:59.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:49:59.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:00.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:00.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:00.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:00.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:00.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:00.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:00.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:00.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:01.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:01.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:01.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:01.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:01.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:01.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:01.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:01.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:02.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:02.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:02.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:02.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:02.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:02.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:02.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:02.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:03.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:03.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:03.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:03.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:03.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:03.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:03.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:03.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:04.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:04.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:04.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:04.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:04.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:04.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:04.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:04.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:05.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:05.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:05.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:05.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:05.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:05.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:05.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:05.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:06.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:06.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:06.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:06.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:06.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:06.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:06.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:06.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:07.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:07.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:07.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:07.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:07.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:07.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:07.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:07.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:08.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:08.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:08.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:08.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:08.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:08.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:08.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:08.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:09.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:09.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:09.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:09.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:09.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:09.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:09.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:09.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:10.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:10.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:10.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:10.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:10.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:10.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:10.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:10.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:11.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:11.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:11.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:11.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:11.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:11.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:11.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:11.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:12.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:12.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:12.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:12.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:12.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:12.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:12.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:12.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:13.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:13.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:13.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:13.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:13.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:13.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:14.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:14.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:14.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:14.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:14.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:14.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:14.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:14.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:14.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:15.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:15.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:15.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:15.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:15.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:15.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:15.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:16.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:16.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:16.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:16.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:16.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:16.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:16.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:16.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:16.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:16.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:17.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:17.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:17.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:17.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:17.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:17.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:18.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:18.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:18.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:18.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:18.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:18.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:18.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:18.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:19.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:19.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:19.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:19.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:19.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:19.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:19.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:19.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:20.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:20.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:20.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:20.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:20.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:20.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:20.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:20.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:21.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:21.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:21.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:21.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:21.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:21.537 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:21.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:21.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:22.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:22.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:22.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:22.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:22.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:22.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:22.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:22.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:23.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:23.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:23.457 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:23.457 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:23.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:23.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:23.935 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:23.935 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:24.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:24.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:24.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:24.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:24.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:24.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:24.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:24.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:25.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:25.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:25.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:25.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:25.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:25.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:25.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:25.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:26.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:26.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:26.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:26.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:26.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:26.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:26.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:26.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:27.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:27.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:27.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:27.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:27.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:27.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:27.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:27.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:28.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:28.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:28.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:28.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:28.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:28.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:28.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:28.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:29.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:29.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:29.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:29.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:29.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:29.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:29.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:29.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:30.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:30.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:30.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:30.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:30.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:30.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:30.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:30.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:31.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:31.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:31.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:31.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:31.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:31.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:31.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:31.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:32.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:32.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:32.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:32.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:32.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:32.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:32.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:32.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:33.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:33.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:33.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:33.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:33.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:33.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:33.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:33.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:34.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:34.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:34.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:34.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:34.567 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:34.567 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:34.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:34.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:35.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:35.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:35.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:35.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:35.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:35.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:36.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:36.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:36.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:36.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:36.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:36.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:36.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:36.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:37.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:37.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:37.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:37.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:37.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:37.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:37.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:37.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:38.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:38.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:38.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:38.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:38.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:38.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:38.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:38.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:39.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:39.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:39.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:39.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:39.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:39.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:39.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:40.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:40.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:40.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:40.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:40.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:40.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:40.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:40.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:41.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:41.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:41.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:41.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:41.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:41.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:41.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:41.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:42.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:42.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:42.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:42.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:42.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:42.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:42.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:42.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:43.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:43.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:43.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:43.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:43.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:43.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:43.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:43.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:44.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:44.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:44.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:44.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:44.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:44.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:45.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:45.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:45.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:45.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:45.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:50:45.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2024-09-15T21:50:45.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2024-09-15T21:50:45.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T21:50:45.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T21:50:45.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T21:50:45.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T21:50:45.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T21:50:45.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T21:50:45.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj250 2024-09-15T21:50:45.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T21:50:45.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T21:50:45.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T21:50:45.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T21:50:45.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T21:50:45.773 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj250 2024-09-15T21:50:45.773 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T21:50:45.773 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj250 2024-09-15T21:50:45.773 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-15T21:50:45.773 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:50:46.272 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T21:50:46.272 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:50:46.272 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:50:46.272 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:50:46.273 INFO:tasks.workunit.client.0.smithi104.stderr:2147483647' 2024-09-15T21:50:46.273 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-15T21:50:46.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-15T21:50:46.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T21:50:46.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-15T21:50:46.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T21:50:46.273 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T21:50:46.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T21:50:46.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T21:50:46.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T21:50:46.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-15T21:50:46.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T21:50:46.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T21:50:46.275 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T21:50:46.275 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-15T21:50:46.275 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T21:50:46.275 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T21:50:46.276 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:50:46.276 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:46.276 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:50:46.276 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T21:50:46.276 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T21:50:46.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T21:50:46.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T21:50:46.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T21:50:46.395 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-15T21:50:46.395 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T21:50:46.395 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T21:50:46.395 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:50:46.396 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:46.396 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:50:46.396 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T21:50:46.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T21:50:46.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2024-09-15T21:50:46.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:565: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2024-09-15T21:50:46.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T21:50:46.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T21:50:46.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T21:50:46.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T21:50:46.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T21:50:46.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T21:50:46.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj250 2024-09-15T21:50:46.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T21:50:46.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T21:50:46.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T21:50:46.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-15T21:50:46.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T21:50:46.515 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj250 2024-09-15T21:50:46.515 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T21:50:46.515 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj250 2024-09-15T21:50:46.515 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-15T21:50:46.515 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:50:46.929 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T21:50:46.930 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:50:46.930 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:50:46.930 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:50:46.930 INFO:tasks.workunit.client.0.smithi104.stderr:2147483647' 2024-09-15T21:50:46.930 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-15T21:50:46.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-15T21:50:46.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T21:50:46.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-15T21:50:46.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T21:50:46.930 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-15T21:50:46.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T21:50:46.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T21:50:46.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T21:50:46.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T21:50:46.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T21:50:46.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T21:50:46.932 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T21:50:46.932 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T21:50:46.932 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T21:50:46.933 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T21:50:46.933 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:50:46.933 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:46.933 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:50:46.933 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.1.asok 2024-09-15T21:50:46.933 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T21:50:47.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T21:50:47.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T21:50:47.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T21:50:47.039 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-15T21:50:47.040 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T21:50:47.040 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T21:50:47.040 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:50:47.040 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:47.040 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:50:47.040 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.1.asok 2024-09-15T21:50:47.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T21:50:47.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2024-09-15T21:50:47.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-15T21:50:47.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T21:50:47.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:50:47.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T21:50:47.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:50:47.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:50:47.152 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:50:47.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:50:47.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T21:50:47.154 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T21:50:47.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T21:50:47.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:50:47.159 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T21:50:47.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T21:50:47.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:50:47.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:50:47.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:50:47.177 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:50:47.172+0000 7f6da411d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:47.209 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:50:47.204+0000 7f6da411d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:47.211 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:50:47.208+0000 7f6da411d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:47.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T21:50:47.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:50:47.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:50:47.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:50:47.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:50:47.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:47.606 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:50:47.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:50:47.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:47.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:50:48.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:48.454 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:50:48.448+0000 7f6da411d7c0 -1 Falling back to public interface 2024-09-15T21:50:49.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:49.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:49.071 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:50:49.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:50:49.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:49.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:50:49.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:50.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:50.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:50.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:50:50.552 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:50:50.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:50.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:50:50.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:51.877 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:50:51.872+0000 7f6da411d7c0 -1 osd.2 48 log_to_monitors true 2024-09-15T21:50:51.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:51.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:51.977 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:50:51.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:50:51.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:51.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:50:52.482 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up out weight 0 up_from 56 up_thru 22 down_at 49 last_clean_interval [20,48) [v2:127.0.0.1:6818/1276066592,v1:127.0.0.1:6819/1276066592] [v2:127.0.0.1:6820/1276066592,v1:127.0.0.1:6821/1276066592] exists,up 8a6b43e9-5157-4016-a80c-69cfcd431874 2024-09-15T21:50:52.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:50:52.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:50:52.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:50:52.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph osd in 2 2024-09-15T21:50:53.247 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 is already in. 2024-09-15T21:50:53.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: sleep 15 2024-09-15T21:51:08.266 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: seq 1 240 2024-09-15T21:51:08.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T21:51:08.267 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T21:51:08.267 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T21:51:08.267 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T21:51:08.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T21:51:08.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T21:51:08.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:08.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T21:51:08.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:08.713 INFO:tasks.workunit.client.0.smithi104.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:08.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T21:51:08.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:08.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-15T21:51:09.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T21:51:09.715 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T21:51:09.715 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T21:51:09.715 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T21:51:09.715 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T21:51:09.716 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T21:51:10.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:10.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:10.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T21:51:10.167 INFO:tasks.workunit.client.0.smithi104.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:10.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T21:51:10.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:10.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-15T21:51:11.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T21:51:11.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T21:51:11.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T21:51:11.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T21:51:11.171 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T21:51:11.171 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T21:51:11.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:11.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:11.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T21:51:11.605 INFO:tasks.workunit.client.0.smithi104.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:11.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T21:51:11.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:11.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-15T21:51:12.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T21:51:12.607 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T21:51:12.607 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T21:51:12.607 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T21:51:12.607 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T21:51:12.608 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T21:51:13.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:13.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:13.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T21:51:13.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T21:51:13.048 INFO:tasks.workunit.client.0.smithi104.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:13.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:13.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-15T21:51:14.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T21:51:14.050 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T21:51:14.050 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T21:51:14.050 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T21:51:14.051 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T21:51:14.051 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T21:51:14.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:14.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T21:51:14.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:14.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T21:51:14.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:14.493 INFO:tasks.workunit.client.0.smithi104.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:14.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-15T21:51:15.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T21:51:15.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T21:51:15.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T21:51:15.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T21:51:15.496 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T21:51:15.496 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T21:51:15.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:15.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:15.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T21:51:15.983 INFO:tasks.workunit.client.0.smithi104.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:15.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T21:51:15.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:15.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-15T21:51:16.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T21:51:16.985 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T21:51:16.986 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T21:51:16.986 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T21:51:16.986 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T21:51:16.986 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T21:51:17.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:17.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:17.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T21:51:17.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T21:51:17.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:17.413 INFO:tasks.workunit.client.0.smithi104.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:17.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-15T21:51:18.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T21:51:18.420 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T21:51:18.420 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T21:51:18.421 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T21:51:18.421 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T21:51:18.421 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T21:51:18.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:18.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:18.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T21:51:18.836 INFO:tasks.workunit.client.0.smithi104.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:18.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T21:51:18.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T21:51:18.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-15T21:51:19.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T21:51:19.838 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T21:51:19.838 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T21:51:19.838 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T21:51:19.839 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T21:51:19.839 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T21:51:20.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-09-15T21:51:20.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-09-15T21:51:20.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T21:51:20.258 INFO:tasks.workunit.client.0.smithi104.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-09-15T21:51:20.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2024-09-15T21:51:20.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: break 2024-09-15T21:51:20.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:582: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-15T21:51:20.665 INFO:tasks.workunit.client.0.smithi104.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T21:51:20.665 INFO:tasks.workunit.client.0.smithi104.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-09-15T21:51:15.809433+0000 54'301 60:901 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2024-09-15T21:49:17.512572+0000 0'0 2024-09-15T21:49:17.512572+0000 0 0 no scrub is scheduled 0 0 2024-09-15T21:51:20.665 INFO:tasks.workunit.client.0.smithi104.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:50:54.420362+0000 0'0 60:60 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-16T22:32:58.876965+0000 0 0 2024-09-15T21:51:20.665 INFO:tasks.workunit.client.0.smithi104.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:49:17.007547+0000 0'0 60:136 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T08:05:50.396986+0000 0 0 2024-09-15T21:51:20.665 INFO:tasks.workunit.client.0.smithi104.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:49:17.191647+0000 0'0 60:42 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T06:23:23.885703+0000 0 0 2024-09-15T21:51:20.665 INFO:tasks.workunit.client.0.smithi104.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:50:54.419898+0000 0'0 60:113 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T03:02:07.592017+0000 0 0 2024-09-15T21:51:20.665 INFO:tasks.workunit.client.0.smithi104.stdout: 2024-09-15T21:51:20.665 INFO:tasks.workunit.client.0.smithi104.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T21:51:20.666 INFO:tasks.workunit.client.0.smithi104.stderr:dumped pgs 2024-09-15T21:51:20.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:583: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T21:51:20.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:51:20.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:51:20.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:51:20.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:51:20.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:51:20.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:51:20.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: sleep 5 2024-09-15T21:51:25.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:586: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-15T21:51:26.419 INFO:tasks.workunit.client.0.smithi104.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T21:51:26.419 INFO:tasks.workunit.client.0.smithi104.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2024-09-15T21:51:21.736286+0000 54'301 63:915 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-09-15T21:49:17.512572+0000 0'0 2024-09-15T21:49:17.512572+0000 0 0 no scrub is scheduled 0 0 2024-09-15T21:51:26.420 INFO:tasks.workunit.client.0.smithi104.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:50:54.420362+0000 0'0 60:60 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-16T22:32:58.876965+0000 0 0 2024-09-15T21:51:26.420 INFO:tasks.workunit.client.0.smithi104.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:49:17.007547+0000 0'0 60:136 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T08:05:50.396986+0000 0 0 2024-09-15T21:51:26.420 INFO:tasks.workunit.client.0.smithi104.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T21:49:17.191647+0000 0'0 60:42 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 periodic scrub scheduled @ 2024-09-17T06:23:23.885703+0000 0 0 2024-09-15T21:51:26.420 INFO:tasks.workunit.client.0.smithi104.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-15T21:51:21.384656+0000 0'0 62:116 [1,0] 1 [1,0] 1 0'0 2024-09-15T21:46:16.597696+0000 0'0 2024-09-15T21:46:16.597696+0000 0 0 no scrub is scheduled 0 0 2024-09-15T21:51:26.420 INFO:tasks.workunit.client.0.smithi104.stdout: 2024-09-15T21:51:26.420 INFO:tasks.workunit.client.0.smithi104.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T21:51:26.420 INFO:tasks.workunit.client.0.smithi104.stderr:dumped pgs 2024-09-15T21:51:26.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-15T21:51:26.621 INFO:tasks.workunit.client.0.smithi104.stdout:{ 2024-09-15T21:51:26.621 INFO:tasks.workunit.client.0.smithi104.stdout: "num_missing": 1, 2024-09-15T21:51:26.621 INFO:tasks.workunit.client.0.smithi104.stdout: "num_unfound": 1, 2024-09-15T21:51:26.621 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": [ 2024-09-15T21:51:26.621 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.621 INFO:tasks.workunit.client.0.smithi104.stdout: "oid": { 2024-09-15T21:51:26.621 INFO:tasks.workunit.client.0.smithi104.stdout: "oid": "obj250", 2024-09-15T21:51:26.621 INFO:tasks.workunit.client.0.smithi104.stdout: "key": "", 2024-09-15T21:51:26.622 INFO:tasks.workunit.client.0.smithi104.stdout: "snapid": -2, 2024-09-15T21:51:26.622 INFO:tasks.workunit.client.0.smithi104.stdout: "hash": 2249616407, 2024-09-15T21:51:26.622 INFO:tasks.workunit.client.0.smithi104.stdout: "max": 0, 2024-09-15T21:51:26.622 INFO:tasks.workunit.client.0.smithi104.stdout: "pool": 2, 2024-09-15T21:51:26.622 INFO:tasks.workunit.client.0.smithi104.stdout: "namespace": "" 2024-09-15T21:51:26.622 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.622 INFO:tasks.workunit.client.0.smithi104.stdout: "need": "54'251", 2024-09-15T21:51:26.622 INFO:tasks.workunit.client.0.smithi104.stdout: "have": "0'0", 2024-09-15T21:51:26.622 INFO:tasks.workunit.client.0.smithi104.stdout: "flags": "none", 2024-09-15T21:51:26.622 INFO:tasks.workunit.client.0.smithi104.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:51:26.622 INFO:tasks.workunit.client.0.smithi104.stdout: "locations": [ 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: ] 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "NotRecovering", 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: "available_might_have_unfound": true, 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: "might_have_unfound": [ 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: "osd": "2(4)", 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: "status": "osd is down" 2024-09-15T21:51:26.623 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.624 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.624 INFO:tasks.workunit.client.0.smithi104.stdout: "more": false 2024-09-15T21:51:26.624 INFO:tasks.workunit.client.0.smithi104.stdout:} 2024-09-15T21:51:26.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:588: TEST_ec_backfill_unfound: ceph pg 2.0 query 2024-09-15T21:51:26.780 INFO:tasks.workunit.client.0.smithi104.stdout:{ 2024-09-15T21:51:26.781 INFO:tasks.workunit.client.0.smithi104.stdout: "snap_trimq": "[]", 2024-09-15T21:51:26.781 INFO:tasks.workunit.client.0.smithi104.stdout: "snap_trimq_len": 0, 2024-09-15T21:51:26.781 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-15T21:51:26.781 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch": 63, 2024-09-15T21:51:26.781 INFO:tasks.workunit.client.0.smithi104.stdout: "up": [ 2024-09-15T21:51:26.781 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T21:51:26.781 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T21:51:26.781 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T21:51:26.781 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T21:51:26.781 INFO:tasks.workunit.client.0.smithi104.stdout: 2147483647 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: "acting": [ 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: 2147483647 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: "acting_recovery_backfill": [ 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: "1(1)", 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: "3(0)", 2024-09-15T21:51:26.782 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: "info": { 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: "pgid": "2.0s0", 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: "shared": "0", 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: "last_update": "54'301", 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: "last_complete": "54'250", 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: "log_tail": "54'200", 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: "last_user_version": 301, 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: "last_backfill": "MAX", 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [], 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: "history": { 2024-09-15T21:51:26.783 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_created": 46, 2024-09-15T21:51:26.784 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_pool_created": 46, 2024-09-15T21:51:26.784 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 62, 2024-09-15T21:51:26.784 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_started": 61, 2024-09-15T21:51:26.784 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 47, 2024-09-15T21:51:26.784 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_clean": 46, 2024-09-15T21:51:26.784 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_split": 0, 2024-09-15T21:51:26.784 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_marked_full": 0, 2024-09-15T21:51:26.784 INFO:tasks.workunit.client.0.smithi104.stdout: "same_up_since": 61, 2024-09-15T21:51:26.784 INFO:tasks.workunit.client.0.smithi104.stdout: "same_interval_since": 61, 2024-09-15T21:51:26.784 INFO:tasks.workunit.client.0.smithi104.stdout: "same_primary_since": 46, 2024-09-15T21:51:26.784 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: "prior_readable_until_ub": 0 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: "stats": { 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: "version": "54'301", 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_seq": 916, 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_epoch": 63, 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-15T21:51:26.785 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fresh": "2024-09-15T21:51:22.222898+0000", 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "last_change": "2024-09-15T21:51:21.736286+0000", 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "last_active": "2024-09-15T21:51:22.222898+0000", 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "last_peered": "2024-09-15T21:51:22.222898+0000", 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean": "2024-09-15T21:49:26.153342+0000", 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_active": "2024-09-15T21:51:21.722862+0000", 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_peered": "2024-09-15T21:51:21.722862+0000", 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "last_unstale": "2024-09-15T21:51:22.222898+0000", 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "last_undegraded": "2024-09-15T21:51:21.384658+0000", 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fullsized": "2024-09-15T21:51:21.383910+0000", 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "mapping_epoch": 61, 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "log_start": "54'200", 2024-09-15T21:51:26.786 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_start": "54'200", 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "created": 46, 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 47, 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "parent": "0.0", 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "parent_split_bits": 0, 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_scrubbed": 0, 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "log_size": 101, 2024-09-15T21:51:26.787 INFO:tasks.workunit.client.0.smithi104.stdout: "log_dups_size": 200, 2024-09-15T21:51:26.788 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_size": 101, 2024-09-15T21:51:26.788 INFO:tasks.workunit.client.0.smithi104.stdout: "stats_invalid": false, 2024-09-15T21:51:26.788 INFO:tasks.workunit.client.0.smithi104.stdout: "dirty_stats_invalid": false, 2024-09-15T21:51:26.788 INFO:tasks.workunit.client.0.smithi104.stdout: "omap_stats_invalid": false, 2024-09-15T21:51:26.788 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_stats_invalid": false, 2024-09-15T21:51:26.788 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T21:51:26.788 INFO:tasks.workunit.client.0.smithi104.stdout: "pin_stats_invalid": false, 2024-09-15T21:51:26.788 INFO:tasks.workunit.client.0.smithi104.stdout: "manifest_stats_invalid": false, 2024-09-15T21:51:26.789 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrimq_len": 0, 2024-09-15T21:51:26.789 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_duration": 0, 2024-09-15T21:51:26.789 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_schedule": "no scrub is scheduled", 2024-09-15T21:51:26.789 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_duration": 0, 2024-09-15T21:51:26.789 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_trimmed": 0, 2024-09-15T21:51:26.789 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrim_duration": 0, 2024-09-15T21:51:26.789 INFO:tasks.workunit.client.0.smithi104.stdout: "stat_sum": { 2024-09-15T21:51:26.789 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes": 1232896, 2024-09-15T21:51:26.789 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects": 301, 2024-09-15T21:51:26.789 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_clones": 0, 2024-09-15T21:51:26.789 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_copies": 1505, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing_on_primary": 1, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing": 1, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_degraded": 303, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_misplaced": 0, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_unfound": 1, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_dirty": 301, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_whiteouts": 0, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read": 0, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read_kb": 0, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write": 301, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write_kb": 1204, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_scrub_errors": 0, 2024-09-15T21:51:26.790 INFO:tasks.workunit.client.0.smithi104.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T21:51:26.791 INFO:tasks.workunit.client.0.smithi104.stdout: "num_deep_scrub_errors": 0, 2024-09-15T21:51:26.791 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_recovered": 0, 2024-09-15T21:51:26.791 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_recovered": 0, 2024-09-15T21:51:26.791 INFO:tasks.workunit.client.0.smithi104.stdout: "num_keys_recovered": 0, 2024-09-15T21:51:26.792 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_omap": 0, 2024-09-15T21:51:26.792 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T21:51:26.792 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T21:51:26.792 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush": 0, 2024-09-15T21:51:26.793 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_kb": 0, 2024-09-15T21:51:26.795 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict": 0, 2024-09-15T21:51:26.796 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_kb": 0, 2024-09-15T21:51:26.796 INFO:tasks.workunit.client.0.smithi104.stdout: "num_promote": 0, 2024-09-15T21:51:26.796 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_high": 0, 2024-09-15T21:51:26.796 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_low": 0, 2024-09-15T21:51:26.796 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_some": 0, 2024-09-15T21:51:26.796 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_full": 0, 2024-09-15T21:51:26.796 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_pinned": 0, 2024-09-15T21:51:26.796 INFO:tasks.workunit.client.0.smithi104.stdout: "num_legacy_snapsets": 0, 2024-09-15T21:51:26.796 INFO:tasks.workunit.client.0.smithi104.stdout: "num_large_omap_objects": 0, 2024-09-15T21:51:26.796 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_manifest": 0, 2024-09-15T21:51:26.796 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_bytes": 0, 2024-09-15T21:51:26.797 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_keys": 0, 2024-09-15T21:51:26.797 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_repaired": 0 2024-09-15T21:51:26.797 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.797 INFO:tasks.workunit.client.0.smithi104.stdout: "up": [ 2024-09-15T21:51:26.797 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T21:51:26.797 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T21:51:26.797 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T21:51:26.797 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T21:51:26.797 INFO:tasks.workunit.client.0.smithi104.stdout: 2147483647 2024-09-15T21:51:26.797 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.797 INFO:tasks.workunit.client.0.smithi104.stdout: "acting": [ 2024-09-15T21:51:26.798 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T21:51:26.798 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T21:51:26.798 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T21:51:26.798 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T21:51:26.798 INFO:tasks.workunit.client.0.smithi104.stdout: 2147483647 2024-09-15T21:51:26.798 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.798 INFO:tasks.workunit.client.0.smithi104.stdout: "avail_no_missing": [ 2024-09-15T21:51:26.798 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T21:51:26.798 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T21:51:26.799 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.799 INFO:tasks.workunit.client.0.smithi104.stdout: "object_location_counts": [ 2024-09-15T21:51:26.799 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.799 INFO:tasks.workunit.client.0.smithi104.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T21:51:26.799 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": 300 2024-09-15T21:51:26.799 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.799 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.799 INFO:tasks.workunit.client.0.smithi104.stdout: "shards": "0(3),4(2)", 2024-09-15T21:51:26.800 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": 1 2024-09-15T21:51:26.800 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.800 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.800 INFO:tasks.workunit.client.0.smithi104.stdout: "blocked_by": [], 2024-09-15T21:51:26.800 INFO:tasks.workunit.client.0.smithi104.stdout: "up_primary": 3, 2024-09-15T21:51:26.800 INFO:tasks.workunit.client.0.smithi104.stdout: "acting_primary": 3, 2024-09-15T21:51:26.800 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [] 2024-09-15T21:51:26.800 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.800 INFO:tasks.workunit.client.0.smithi104.stdout: "empty": 0, 2024-09-15T21:51:26.800 INFO:tasks.workunit.client.0.smithi104.stdout: "dne": 0, 2024-09-15T21:51:26.800 INFO:tasks.workunit.client.0.smithi104.stdout: "incomplete": 0, 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 62, 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: "hit_set_history": { 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: "current_last_update": "0'0", 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: "history": [] 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: "peer_info": [ 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: "peer": "0(3)", 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: "pgid": "2.0s3", 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: "shared": "3", 2024-09-15T21:51:26.801 INFO:tasks.workunit.client.0.smithi104.stdout: "last_update": "54'301", 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "last_complete": "54'301", 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "log_tail": "54'200", 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "last_user_version": 301, 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "last_backfill": "MAX", 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [], 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "history": { 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_created": 46, 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_pool_created": 46, 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 62, 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_started": 61, 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 47, 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_clean": 46, 2024-09-15T21:51:26.802 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_split": 0, 2024-09-15T21:51:26.803 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_marked_full": 0, 2024-09-15T21:51:26.803 INFO:tasks.workunit.client.0.smithi104.stdout: "same_up_since": 61, 2024-09-15T21:51:26.803 INFO:tasks.workunit.client.0.smithi104.stdout: "same_interval_since": 61, 2024-09-15T21:51:26.803 INFO:tasks.workunit.client.0.smithi104.stdout: "same_primary_since": 46, 2024-09-15T21:51:26.803 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T21:51:26.803 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.803 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T21:51:26.803 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.803 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.803 INFO:tasks.workunit.client.0.smithi104.stdout: "prior_readable_until_ub": 0 2024-09-15T21:51:26.803 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "stats": { 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "version": "54'301", 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_seq": 904, 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_epoch": 61, 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "active+undersized+degraded", 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fresh": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "last_change": "2024-09-15T21:49:28.454387+0000", 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "last_active": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "last_peered": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean": "2024-09-15T21:49:26.153342+0000", 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_active": "2024-09-15T21:49:28.454387+0000", 2024-09-15T21:51:26.804 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_peered": "2024-09-15T21:49:28.454387+0000", 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "last_unstale": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "last_undegraded": "2024-09-15T21:49:28.154831+0000", 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fullsized": "2024-09-15T21:49:28.154622+0000", 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "mapping_epoch": 61, 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "log_start": "54'200", 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_start": "54'200", 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "created": 46, 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 47, 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "parent": "0.0", 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "parent_split_bits": 0, 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T21:51:26.805 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_scrubbed": 0, 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "log_size": 101, 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "log_dups_size": 200, 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_size": 101, 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "stats_invalid": false, 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "dirty_stats_invalid": false, 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "omap_stats_invalid": false, 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_stats_invalid": false, 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T21:51:26.806 INFO:tasks.workunit.client.0.smithi104.stdout: "pin_stats_invalid": false, 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "manifest_stats_invalid": false, 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrimq_len": 0, 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_duration": 0, 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T22:12:59.587117+0000", 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_duration": 0, 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_trimmed": 0, 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrim_duration": 0, 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "stat_sum": { 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes": 1232896, 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects": 301, 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_clones": 0, 2024-09-15T21:51:26.807 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_copies": 1505, 2024-09-15T21:51:26.808 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T21:51:26.808 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing": 0, 2024-09-15T21:51:26.808 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_degraded": 301, 2024-09-15T21:51:26.808 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_misplaced": 0, 2024-09-15T21:51:26.808 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_unfound": 0, 2024-09-15T21:51:26.808 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_dirty": 301, 2024-09-15T21:51:26.808 INFO:tasks.workunit.client.0.smithi104.stdout: "num_whiteouts": 0, 2024-09-15T21:51:26.808 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read": 0, 2024-09-15T21:51:26.808 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read_kb": 0, 2024-09-15T21:51:26.808 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write": 301, 2024-09-15T21:51:26.808 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write_kb": 1204, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_scrub_errors": 0, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_deep_scrub_errors": 0, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_recovered": 0, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_recovered": 0, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_keys_recovered": 0, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_omap": 0, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush": 0, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_kb": 0, 2024-09-15T21:51:26.809 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_kb": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_promote": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_high": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_low": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_some": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_full": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_pinned": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_legacy_snapsets": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_large_omap_objects": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_manifest": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_bytes": 0, 2024-09-15T21:51:26.810 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_keys": 0, 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_repaired": 0 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: "up": [ 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: 2147483647 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: "acting": [ 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T21:51:26.811 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: 2147483647 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: "avail_no_missing": [ 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: "3(0)", 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: "1(1)", 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: "object_location_counts": [ 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.812 INFO:tasks.workunit.client.0.smithi104.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T21:51:26.813 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": 301 2024-09-15T21:51:26.813 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.813 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.813 INFO:tasks.workunit.client.0.smithi104.stdout: "blocked_by": [], 2024-09-15T21:51:26.813 INFO:tasks.workunit.client.0.smithi104.stdout: "up_primary": 3, 2024-09-15T21:51:26.813 INFO:tasks.workunit.client.0.smithi104.stdout: "acting_primary": 3, 2024-09-15T21:51:26.813 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [] 2024-09-15T21:51:26.813 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.813 INFO:tasks.workunit.client.0.smithi104.stdout: "empty": 0, 2024-09-15T21:51:26.813 INFO:tasks.workunit.client.0.smithi104.stdout: "dne": 0, 2024-09-15T21:51:26.813 INFO:tasks.workunit.client.0.smithi104.stdout: "incomplete": 0, 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 62, 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: "hit_set_history": { 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: "current_last_update": "0'0", 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: "history": [] 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: "peer": "1(1)", 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: "pgid": "2.0s1", 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: "shared": "1", 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: "last_update": "54'301", 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: "last_complete": "54'301", 2024-09-15T21:51:26.814 INFO:tasks.workunit.client.0.smithi104.stdout: "log_tail": "54'200", 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "last_user_version": 301, 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "last_backfill": "MAX", 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [], 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "history": { 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_created": 46, 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_pool_created": 46, 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 62, 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_started": 61, 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 47, 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_clean": 46, 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_split": 0, 2024-09-15T21:51:26.815 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_marked_full": 0, 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: "same_up_since": 61, 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: "same_interval_since": 61, 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: "same_primary_since": 46, 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: "prior_readable_until_ub": 0 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: "stats": { 2024-09-15T21:51:26.816 INFO:tasks.workunit.client.0.smithi104.stdout: "version": "54'301", 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_seq": 335, 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_epoch": 54, 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "active+undersized+degraded", 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fresh": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "last_change": "2024-09-15T21:49:28.454387+0000", 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "last_active": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "last_peered": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean": "2024-09-15T21:49:26.153342+0000", 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_active": "2024-09-15T21:49:28.454387+0000", 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_peered": "2024-09-15T21:49:28.454387+0000", 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "last_unstale": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "last_undegraded": "2024-09-15T21:49:28.154831+0000", 2024-09-15T21:51:26.817 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fullsized": "2024-09-15T21:49:28.154622+0000", 2024-09-15T21:51:26.818 INFO:tasks.workunit.client.0.smithi104.stdout: "mapping_epoch": 61, 2024-09-15T21:51:26.818 INFO:tasks.workunit.client.0.smithi104.stdout: "log_start": "54'200", 2024-09-15T21:51:26.818 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_start": "54'200", 2024-09-15T21:51:26.818 INFO:tasks.workunit.client.0.smithi104.stdout: "created": 46, 2024-09-15T21:51:26.818 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 47, 2024-09-15T21:51:26.818 INFO:tasks.workunit.client.0.smithi104.stdout: "parent": "0.0", 2024-09-15T21:51:26.818 INFO:tasks.workunit.client.0.smithi104.stdout: "parent_split_bits": 0, 2024-09-15T21:51:26.818 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T21:51:26.819 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.819 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T21:51:26.819 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.819 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.819 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_scrubbed": 0, 2024-09-15T21:51:26.819 INFO:tasks.workunit.client.0.smithi104.stdout: "log_size": 101, 2024-09-15T21:51:26.820 INFO:tasks.workunit.client.0.smithi104.stdout: "log_dups_size": 200, 2024-09-15T21:51:26.820 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_size": 101, 2024-09-15T21:51:26.820 INFO:tasks.workunit.client.0.smithi104.stdout: "stats_invalid": false, 2024-09-15T21:51:26.820 INFO:tasks.workunit.client.0.smithi104.stdout: "dirty_stats_invalid": false, 2024-09-15T21:51:26.820 INFO:tasks.workunit.client.0.smithi104.stdout: "omap_stats_invalid": false, 2024-09-15T21:51:26.820 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_stats_invalid": false, 2024-09-15T21:51:26.820 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T21:51:26.820 INFO:tasks.workunit.client.0.smithi104.stdout: "pin_stats_invalid": false, 2024-09-15T21:51:26.820 INFO:tasks.workunit.client.0.smithi104.stdout: "manifest_stats_invalid": false, 2024-09-15T21:51:26.820 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrimq_len": 0, 2024-09-15T21:51:26.820 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_duration": 0, 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T22:12:59.587117+0000", 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_duration": 0, 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_trimmed": 0, 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrim_duration": 0, 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "stat_sum": { 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes": 1232896, 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects": 301, 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_clones": 0, 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_copies": 1505, 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing": 1, 2024-09-15T21:51:26.821 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_degraded": 301, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_misplaced": 0, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_unfound": 0, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_dirty": 301, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_whiteouts": 0, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read": 0, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read_kb": 0, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write": 301, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write_kb": 1204, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_scrub_errors": 0, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_deep_scrub_errors": 0, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_recovered": 0, 2024-09-15T21:51:26.822 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_recovered": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_keys_recovered": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_omap": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_kb": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_kb": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_promote": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_high": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_low": 0, 2024-09-15T21:51:26.823 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_some": 0, 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_full": 0, 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_pinned": 0, 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: "num_legacy_snapsets": 0, 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: "num_large_omap_objects": 0, 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_manifest": 0, 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_bytes": 0, 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_keys": 0, 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_repaired": 0 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: "up": [ 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T21:51:26.824 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: 2147483647 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: "acting": [ 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: 2147483647 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: "avail_no_missing": [ 2024-09-15T21:51:26.825 INFO:tasks.workunit.client.0.smithi104.stdout: "3(0)", 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: "1(1)", 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: "object_location_counts": [ 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": 301 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: "blocked_by": [], 2024-09-15T21:51:26.826 INFO:tasks.workunit.client.0.smithi104.stdout: "up_primary": 3, 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: "acting_primary": 3, 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [] 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: "empty": 0, 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: "dne": 0, 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: "incomplete": 0, 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 62, 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: "hit_set_history": { 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: "current_last_update": "0'0", 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: "history": [] 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.827 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.828 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.828 INFO:tasks.workunit.client.0.smithi104.stdout: "peer": "4(2)", 2024-09-15T21:51:26.828 INFO:tasks.workunit.client.0.smithi104.stdout: "pgid": "2.0s2", 2024-09-15T21:51:26.828 INFO:tasks.workunit.client.0.smithi104.stdout: "shared": "2", 2024-09-15T21:51:26.828 INFO:tasks.workunit.client.0.smithi104.stdout: "last_update": "54'301", 2024-09-15T21:51:26.828 INFO:tasks.workunit.client.0.smithi104.stdout: "last_complete": "54'301", 2024-09-15T21:51:26.828 INFO:tasks.workunit.client.0.smithi104.stdout: "log_tail": "54'200", 2024-09-15T21:51:26.828 INFO:tasks.workunit.client.0.smithi104.stdout: "last_user_version": 301, 2024-09-15T21:51:26.828 INFO:tasks.workunit.client.0.smithi104.stdout: "last_backfill": "MAX", 2024-09-15T21:51:26.828 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [], 2024-09-15T21:51:26.828 INFO:tasks.workunit.client.0.smithi104.stdout: "history": { 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_created": 46, 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_pool_created": 46, 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 62, 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_started": 61, 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 47, 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_clean": 46, 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_split": 0, 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_marked_full": 0, 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "same_up_since": 61, 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "same_interval_since": 61, 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "same_primary_since": 46, 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T21:51:26.829 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: "prior_readable_until_ub": 0 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: "stats": { 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: "version": "54'301", 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_seq": 335, 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_epoch": 54, 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "active+undersized+degraded", 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fresh": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.830 INFO:tasks.workunit.client.0.smithi104.stdout: "last_change": "2024-09-15T21:49:28.454387+0000", 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "last_active": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "last_peered": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean": "2024-09-15T21:49:26.153342+0000", 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_active": "2024-09-15T21:49:28.454387+0000", 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_peered": "2024-09-15T21:49:28.454387+0000", 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "last_unstale": "2024-09-15T21:50:45.765772+0000", 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "last_undegraded": "2024-09-15T21:49:28.154831+0000", 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fullsized": "2024-09-15T21:49:28.154622+0000", 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "mapping_epoch": 61, 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "log_start": "54'200", 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_start": "54'200", 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "created": 46, 2024-09-15T21:51:26.831 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 47, 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "parent": "0.0", 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "parent_split_bits": 0, 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T21:49:17.512572+0000", 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_scrubbed": 0, 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "log_size": 101, 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "log_dups_size": 200, 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_size": 101, 2024-09-15T21:51:26.832 INFO:tasks.workunit.client.0.smithi104.stdout: "stats_invalid": false, 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "dirty_stats_invalid": false, 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "omap_stats_invalid": false, 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_stats_invalid": false, 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "pin_stats_invalid": false, 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "manifest_stats_invalid": false, 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrimq_len": 0, 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_duration": 0, 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T22:12:59.587117+0000", 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_duration": 0, 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_trimmed": 0, 2024-09-15T21:51:26.833 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrim_duration": 0, 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "stat_sum": { 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes": 1232896, 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects": 301, 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_clones": 0, 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_copies": 1505, 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing": 0, 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_degraded": 301, 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_misplaced": 0, 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_unfound": 0, 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_dirty": 301, 2024-09-15T21:51:26.834 INFO:tasks.workunit.client.0.smithi104.stdout: "num_whiteouts": 0, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read": 0, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read_kb": 0, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write": 301, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write_kb": 1204, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_scrub_errors": 0, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_deep_scrub_errors": 0, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_recovered": 0, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_recovered": 0, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_keys_recovered": 0, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_omap": 0, 2024-09-15T21:51:26.835 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_kb": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_kb": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_promote": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_high": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_low": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_some": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_full": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_pinned": 0, 2024-09-15T21:51:26.836 INFO:tasks.workunit.client.0.smithi104.stdout: "num_legacy_snapsets": 0, 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: "num_large_omap_objects": 0, 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_manifest": 0, 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_bytes": 0, 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_keys": 0, 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_repaired": 0 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: "up": [ 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T21:51:26.837 INFO:tasks.workunit.client.0.smithi104.stdout: 2147483647 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: "acting": [ 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: 2147483647 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: "avail_no_missing": [ 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: "3(0)", 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: "1(1)", 2024-09-15T21:51:26.838 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: "object_location_counts": [ 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": 301 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: "blocked_by": [], 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: "up_primary": 3, 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: "acting_primary": 3, 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [] 2024-09-15T21:51:26.839 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: "empty": 0, 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: "dne": 0, 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: "incomplete": 0, 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 62, 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: "hit_set_history": { 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: "current_last_update": "0'0", 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: "history": [] 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: "recovery_state": [ 2024-09-15T21:51:26.840 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: "name": "Started/Primary/Active", 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: "enter_time": "2024-09-15T21:51:21.384020+0000", 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: "might_have_unfound": [ 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: "osd": "0(3)", 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: "status": "already probed" 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: "osd": "1(1)", 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: "status": "already probed" 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.841 INFO:tasks.workunit.client.0.smithi104.stdout: "osd": "2(4)", 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: "status": "osd is down" 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: "osd": "4(2)", 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: "status": "already probed" 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: "recovery_progress": { 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: "backfill_targets": [], 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: "waiting_on_backfill": [], 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: "last_backfill_started": "MIN", 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: "backfill_info": { 2024-09-15T21:51:26.842 INFO:tasks.workunit.client.0.smithi104.stdout: "begin": "MIN", 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: "end": "MIN", 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": [] 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: "peer_backfill_info": [], 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: "backfills_in_flight": [], 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: "recovering": [], 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: "pg_backend": { 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: "recovery_ops": [], 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: "read_ops": [] 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.843 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: "name": "Started", 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: "enter_time": "2024-09-15T21:51:20.922647+0000" 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: "scrubber": { 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: "active": false, 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: "must_scrub": false, 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: "must_deep_scrub": false, 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: "must_repair": false, 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: "need_auto": false, 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_reg_stamp": "2024-09-16T22:12:59.587117+0000", 2024-09-15T21:51:26.844 INFO:tasks.workunit.client.0.smithi104.stdout: "schedule": "not registered for scrubbing" 2024-09-15T21:51:26.845 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T21:51:26.845 INFO:tasks.workunit.client.0.smithi104.stdout: "agent_state": {} 2024-09-15T21:51:26.845 INFO:tasks.workunit.client.0.smithi104.stdout:} 2024-09-15T21:51:26.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:590: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-15T21:51:26.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:590: TEST_ec_backfill_unfound: grep -q obj250 2024-09-15T21:51:26.948 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-15T21:51:26.948 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2024-09-15T21:51:27.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: check=true 2024-09-15T21:51:27.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: test true == true 2024-09-15T21:51:27.093 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-15T21:51:27.093 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2024-09-15T21:51:27.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2024-09-15T21:51:27.286 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: check='osd is down' 2024-09-15T21:51:27.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:596: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2024-09-15T21:51:27.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-15T21:51:27.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2024-09-15T21:51:27.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2024-09-15T21:51:27.452 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: check='2(4)' 2024-09-15T21:51:27.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:599: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2024-09-15T21:51:27.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:601: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-15T21:51:27.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T21:51:27.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:51:27.453 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:51:27.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:51:27.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:51:27.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T21:51:27.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T21:51:27.455 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T21:51:27.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:51:27.456 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T21:51:27.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T21:51:27.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:51:27.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:51:27.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:51:27.476 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:51:27.476+0000 7f60f08107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:27.478 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:51:27.476+0000 7f60f08107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:27.479 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:51:27.480+0000 7f60f08107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:27.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T21:51:27.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:51:27.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:51:27.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:51:27.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:51:27.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:27.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:51:27.949 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:51:27.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:51:27.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:28.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:28.497 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:51:28.495+0000 7f60f08107c0 -1 Falling back to public interface 2024-09-15T21:51:29.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:29.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:29.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:51:29.398 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:51:29.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:29.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:51:29.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:30.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:30.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:30.883 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:51:30.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:51:30.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:30.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:51:31.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:31.640 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:51:31.639+0000 7f60f08107c0 -1 osd.2 60 log_to_monitors true 2024-09-15T21:51:32.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:32.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:32.359 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:51:32.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:51:32.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:32.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:51:32.977 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 65 up_thru 22 down_at 61 last_clean_interval [56,60) [v2:127.0.0.1:6818/3002179617,v1:127.0.0.1:6819/3002179617] [v2:127.0.0.1:6820/3002179617,v1:127.0.0.1:6821/3002179617] exists,up 8a6b43e9-5157-4016-a80c-69cfcd431874 2024-09-15T21:51:32.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:51:32.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:51:32.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:51:32.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-15T21:51:37.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:605: TEST_ec_backfill_unfound: test 124 = 124 2024-09-15T21:51:37.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:607: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-15T21:51:38.278 INFO:tasks.workunit.client.0.smithi104.stderr:pg has no unfound objects 2024-09-15T21:51:38.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_backfill_unfound: wait_for_clean 2024-09-15T21:51:38.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:51:38.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:51:38.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:51:38.300 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:51:38.300 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:51:38.300 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:51:38.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:51:38.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:51:38.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:51:38.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:51:38.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:51:38.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:51:38.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:51:38.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:51:38.414 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:51:38.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:51:38.854 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:51:38.854 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:51:38.854 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:51:38.854 INFO:tasks.workunit.client.0.smithi104.stderr:4' 2024-09-15T21:51:38.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:51:38.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:51:38.855 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:51:39.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803841 2024-09-15T21:51:39.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803841 2024-09-15T21:51:39.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841' 2024-09-15T21:51:39.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:51:39.027 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:51:39.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574909 2024-09-15T21:51:39.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574909 2024-09-15T21:51:39.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574909' 2024-09-15T21:51:39.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:51:39.179 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:51:39.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=279172874244 2024-09-15T21:51:39.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 279172874244 2024-09-15T21:51:39.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574909 2-279172874244' 2024-09-15T21:51:39.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:51:39.340 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:51:39.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018929 2024-09-15T21:51:39.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018929 2024-09-15T21:51:39.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574909 2-279172874244 3-128849018929' 2024-09-15T21:51:39.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:51:39.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:51:39.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659175 2024-09-15T21:51:39.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659175 2024-09-15T21:51:39.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574909 2-279172874244 3-128849018929 4-176093659175' 2024-09-15T21:51:39.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:51:39.802 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803841 2024-09-15T21:51:39.802 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:51:39.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:51:39.805 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803841 2024-09-15T21:51:39.805 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:51:39.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803841 2024-09-15T21:51:39.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803841' 2024-09-15T21:51:39.806 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803841 2024-09-15T21:51:39.806 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:51:40.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803841 -lt 25769803841 2024-09-15T21:51:40.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:51:40.232 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574909 2024-09-15T21:51:40.232 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:51:40.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:51:40.233 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574909 2024-09-15T21:51:40.233 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:51:40.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574909 2024-09-15T21:51:40.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574909' 2024-09-15T21:51:40.235 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574909 2024-09-15T21:51:40.235 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:51:40.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574907 -lt 55834574909 2024-09-15T21:51:40.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:51:41.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:51:41.691 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:51:42.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574909 -lt 55834574909 2024-09-15T21:51:42.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:51:42.210 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-279172874244 2024-09-15T21:51:42.211 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:51:42.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:51:42.213 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-279172874244 2024-09-15T21:51:42.213 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:51:42.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=279172874244 2024-09-15T21:51:42.214 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 279172874244 2024-09-15T21:51:42.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 279172874244' 2024-09-15T21:51:42.214 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:51:42.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 279172874244 -lt 279172874244 2024-09-15T21:51:42.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:51:42.777 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018929 2024-09-15T21:51:42.777 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:51:42.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:51:42.780 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018929 2024-09-15T21:51:42.780 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:51:42.780 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 128849018929 2024-09-15T21:51:42.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018929 2024-09-15T21:51:42.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018929' 2024-09-15T21:51:42.781 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:51:43.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018929 -lt 128849018929 2024-09-15T21:51:43.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:51:43.197 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-176093659175 2024-09-15T21:51:43.197 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:51:43.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:51:43.199 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-176093659175 2024-09-15T21:51:43.199 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:51:43.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659175 2024-09-15T21:51:43.201 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 176093659175 2024-09-15T21:51:43.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 176093659175' 2024-09-15T21:51:43.201 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:51:43.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659176 -lt 176093659175 2024-09-15T21:51:43.733 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:51:43.733 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:51:43.733 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:51:44.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:51:44.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:51:44.305 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:51:44.306 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:51:44.306 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:51:44.306 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:51:44.306 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:51:44.306 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:51:44.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:51:44.711 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:51:44.712 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:51:44.712 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:51:45.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T21:51:45.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T21:51:45.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:51:45.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T21:51:45.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:51:45.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:51:45.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:51:45.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:51:45.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:51:45.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:51:45.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:51:45.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:51:45.361 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:51:45.361 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:51:45.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:51:45.768 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:51:45.768 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:51:45.768 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:51:46.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:51:46.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:51:46.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:51:46.311 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: seq 1 300 2024-09-15T21:51:46.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2024-09-15T21:51:46.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2024-09-15T21:51:46.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2024-09-15T21:51:46.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2024-09-15T21:51:46.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2024-09-15T21:51:46.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2024-09-15T21:51:46.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2024-09-15T21:51:46.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2024-09-15T21:51:46.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2024-09-15T21:51:46.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2024-09-15T21:51:46.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2024-09-15T21:51:46.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2024-09-15T21:51:46.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2024-09-15T21:51:46.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2024-09-15T21:51:46.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2024-09-15T21:51:46.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2024-09-15T21:51:46.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2024-09-15T21:51:46.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2024-09-15T21:51:46.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2024-09-15T21:51:46.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-15T21:51:46.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:46.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:46.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2024-09-15T21:51:46.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2024-09-15T21:51:47.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2024-09-15T21:51:47.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2024-09-15T21:51:47.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2024-09-15T21:51:47.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2024-09-15T21:51:47.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2024-09-15T21:51:47.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2024-09-15T21:51:47.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2024-09-15T21:51:47.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2024-09-15T21:51:47.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2024-09-15T21:51:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2024-09-15T21:51:47.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2024-09-15T21:51:47.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2024-09-15T21:51:47.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2024-09-15T21:51:47.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2024-09-15T21:51:47.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2024-09-15T21:51:47.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2024-09-15T21:51:47.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.539 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2024-09-15T21:51:47.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.568 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2024-09-15T21:51:47.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2024-09-15T21:51:47.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2024-09-15T21:51:47.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2024-09-15T21:51:47.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2024-09-15T21:51:47.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2024-09-15T21:51:47.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2024-09-15T21:51:47.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2024-09-15T21:51:47.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2024-09-15T21:51:47.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2024-09-15T21:51:47.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2024-09-15T21:51:47.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2024-09-15T21:51:47.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2024-09-15T21:51:47.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2024-09-15T21:51:47.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-15T21:51:47.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:47.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:47.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2024-09-15T21:51:47.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.025 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2024-09-15T21:51:48.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2024-09-15T21:51:48.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2024-09-15T21:51:48.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2024-09-15T21:51:48.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2024-09-15T21:51:48.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2024-09-15T21:51:48.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2024-09-15T21:51:48.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2024-09-15T21:51:48.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2024-09-15T21:51:48.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2024-09-15T21:51:48.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2024-09-15T21:51:48.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2024-09-15T21:51:48.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2024-09-15T21:51:48.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2024-09-15T21:51:48.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2024-09-15T21:51:48.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2024-09-15T21:51:48.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2024-09-15T21:51:48.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2024-09-15T21:51:48.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2024-09-15T21:51:48.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2024-09-15T21:51:48.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2024-09-15T21:51:48.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2024-09-15T21:51:48.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2024-09-15T21:51:48.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2024-09-15T21:51:48.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2024-09-15T21:51:48.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2024-09-15T21:51:48.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2024-09-15T21:51:48.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2024-09-15T21:51:48.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2024-09-15T21:51:48.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2024-09-15T21:51:48.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2024-09-15T21:51:48.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2024-09-15T21:51:48.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2024-09-15T21:51:48.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-15T21:51:48.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:48.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:48.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2024-09-15T21:51:48.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2024-09-15T21:51:49.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2024-09-15T21:51:49.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2024-09-15T21:51:49.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2024-09-15T21:51:49.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2024-09-15T21:51:49.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2024-09-15T21:51:49.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2024-09-15T21:51:49.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2024-09-15T21:51:49.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2024-09-15T21:51:49.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2024-09-15T21:51:49.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2024-09-15T21:51:49.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2024-09-15T21:51:49.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2024-09-15T21:51:49.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2024-09-15T21:51:49.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2024-09-15T21:51:49.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2024-09-15T21:51:49.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2024-09-15T21:51:49.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2024-09-15T21:51:49.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2024-09-15T21:51:49.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2024-09-15T21:51:49.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2024-09-15T21:51:49.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2024-09-15T21:51:49.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2024-09-15T21:51:49.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2024-09-15T21:51:49.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2024-09-15T21:51:49.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2024-09-15T21:51:49.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2024-09-15T21:51:49.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2024-09-15T21:51:49.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2024-09-15T21:51:49.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2024-09-15T21:51:49.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2024-09-15T21:51:49.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2024-09-15T21:51:49.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2024-09-15T21:51:49.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:49.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:49.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2024-09-15T21:51:49.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2024-09-15T21:51:50.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2024-09-15T21:51:50.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2024-09-15T21:51:50.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2024-09-15T21:51:50.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2024-09-15T21:51:50.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2024-09-15T21:51:50.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2024-09-15T21:51:50.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2024-09-15T21:51:50.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2024-09-15T21:51:50.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2024-09-15T21:51:50.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2024-09-15T21:51:50.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2024-09-15T21:51:50.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2024-09-15T21:51:50.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2024-09-15T21:51:50.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2024-09-15T21:51:50.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2024-09-15T21:51:50.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2024-09-15T21:51:50.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2024-09-15T21:51:50.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2024-09-15T21:51:50.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2024-09-15T21:51:50.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2024-09-15T21:51:50.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2024-09-15T21:51:50.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2024-09-15T21:51:50.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2024-09-15T21:51:50.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2024-09-15T21:51:50.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2024-09-15T21:51:50.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2024-09-15T21:51:50.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2024-09-15T21:51:50.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2024-09-15T21:51:50.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2024-09-15T21:51:50.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2024-09-15T21:51:50.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2024-09-15T21:51:50.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2024-09-15T21:51:50.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2024-09-15T21:51:50.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:50.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:50.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2024-09-15T21:51:50.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2024-09-15T21:51:51.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2024-09-15T21:51:51.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2024-09-15T21:51:51.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2024-09-15T21:51:51.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2024-09-15T21:51:51.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2024-09-15T21:51:51.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2024-09-15T21:51:51.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2024-09-15T21:51:51.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2024-09-15T21:51:51.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2024-09-15T21:51:51.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2024-09-15T21:51:51.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2024-09-15T21:51:51.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2024-09-15T21:51:51.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2024-09-15T21:51:51.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2024-09-15T21:51:51.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2024-09-15T21:51:51.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2024-09-15T21:51:51.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2024-09-15T21:51:51.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2024-09-15T21:51:51.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2024-09-15T21:51:51.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2024-09-15T21:51:51.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2024-09-15T21:51:51.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2024-09-15T21:51:51.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2024-09-15T21:51:51.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2024-09-15T21:51:51.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2024-09-15T21:51:51.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2024-09-15T21:51:51.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2024-09-15T21:51:51.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2024-09-15T21:51:51.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2024-09-15T21:51:51.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2024-09-15T21:51:51.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2024-09-15T21:51:51.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:51.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:51.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2024-09-15T21:51:51.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2024-09-15T21:51:52.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2024-09-15T21:51:52.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2024-09-15T21:51:52.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2024-09-15T21:51:52.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2024-09-15T21:51:52.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2024-09-15T21:51:52.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2024-09-15T21:51:52.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2024-09-15T21:51:52.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2024-09-15T21:51:52.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2024-09-15T21:51:52.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2024-09-15T21:51:52.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2024-09-15T21:51:52.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2024-09-15T21:51:52.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2024-09-15T21:51:52.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2024-09-15T21:51:52.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2024-09-15T21:51:52.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2024-09-15T21:51:52.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2024-09-15T21:51:52.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2024-09-15T21:51:52.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2024-09-15T21:51:52.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2024-09-15T21:51:52.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2024-09-15T21:51:52.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2024-09-15T21:51:52.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2024-09-15T21:51:52.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2024-09-15T21:51:52.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2024-09-15T21:51:52.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2024-09-15T21:51:52.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2024-09-15T21:51:52.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2024-09-15T21:51:52.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2024-09-15T21:51:52.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2024-09-15T21:51:52.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2024-09-15T21:51:52.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.935 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2024-09-15T21:51:52.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2024-09-15T21:51:52.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2024-09-15T21:51:52.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:52.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:52.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2024-09-15T21:51:52.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2024-09-15T21:51:53.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2024-09-15T21:51:53.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2024-09-15T21:51:53.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2024-09-15T21:51:53.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2024-09-15T21:51:53.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2024-09-15T21:51:53.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2024-09-15T21:51:53.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2024-09-15T21:51:53.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2024-09-15T21:51:53.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2024-09-15T21:51:53.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2024-09-15T21:51:53.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2024-09-15T21:51:53.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2024-09-15T21:51:53.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2024-09-15T21:51:53.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2024-09-15T21:51:53.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2024-09-15T21:51:53.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2024-09-15T21:51:53.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2024-09-15T21:51:53.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2024-09-15T21:51:53.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2024-09-15T21:51:53.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2024-09-15T21:51:53.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2024-09-15T21:51:53.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2024-09-15T21:51:53.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2024-09-15T21:51:53.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2024-09-15T21:51:53.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2024-09-15T21:51:53.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2024-09-15T21:51:53.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2024-09-15T21:51:53.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2024-09-15T21:51:53.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.911 INFO:tasks.workunit.client.0.smithi104.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2024-09-15T21:51:53.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2024-09-15T21:51:53.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2024-09-15T21:51:53.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2024-09-15T21:51:53.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:53.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:53.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2024-09-15T21:51:53.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2024-09-15T21:51:54.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2024-09-15T21:51:54.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2024-09-15T21:51:54.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2024-09-15T21:51:54.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2024-09-15T21:51:54.118 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2024-09-15T21:51:54.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2024-09-15T21:51:54.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2024-09-15T21:51:54.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2024-09-15T21:51:54.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2024-09-15T21:51:54.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2024-09-15T21:51:54.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2024-09-15T21:51:54.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2024-09-15T21:51:54.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2024-09-15T21:51:54.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2024-09-15T21:51:54.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2024-09-15T21:51:54.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2024-09-15T21:51:54.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2024-09-15T21:51:54.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2024-09-15T21:51:54.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2024-09-15T21:51:54.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2024-09-15T21:51:54.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2024-09-15T21:51:54.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2024-09-15T21:51:54.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2024-09-15T21:51:54.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2024-09-15T21:51:54.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2024-09-15T21:51:54.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2024-09-15T21:51:54.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2024-09-15T21:51:54.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2024-09-15T21:51:54.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2024-09-15T21:51:54.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2024-09-15T21:51:54.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2024-09-15T21:51:54.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2024-09-15T21:51:54.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2024-09-15T21:51:54.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2024-09-15T21:51:54.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:54.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:54.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2024-09-15T21:51:54.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.025 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2024-09-15T21:51:55.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2024-09-15T21:51:55.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2024-09-15T21:51:55.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2024-09-15T21:51:55.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2024-09-15T21:51:55.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2024-09-15T21:51:55.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2024-09-15T21:51:55.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2024-09-15T21:51:55.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2024-09-15T21:51:55.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2024-09-15T21:51:55.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2024-09-15T21:51:55.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T21:51:55.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2024-09-15T21:51:55.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2024-09-15T21:51:55.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T21:51:55.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-15T21:51:55.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T21:51:55.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T21:51:56.089 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T21:51:56.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T21:51:56.755 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T21:51:56.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T21:51:56.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T21:51:56.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:51:56.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T21:51:56.774 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:51:56.774 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:51:56.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:51:56.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:51:56.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:51:57.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:51:57.088 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:51:57.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:51:57.090 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:51:57.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:51:57.092 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:51:57.092 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:51:57.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:51:57.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:51:57.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:51:57.094 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:51:57.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:51:57.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:51:57.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T21:51:57.161 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:51:57.162 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:57.162 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:51:57.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T21:51:57.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:51:57.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:51:57.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T21:51:57.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T21:51:57.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T21:51:57.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T21:51:57.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T21:51:57.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:51:57.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T21:51:57.163 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:51:57.163 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:51:57.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:51:57.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:51:57.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:51:57.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:51:57.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:51:57.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:51:57.165 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:51:57.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:51:57.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:51:57.166 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:51:57.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:51:57.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:51:57.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:51:57.167 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:51:57.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:51:57.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:51:57.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T21:51:57.171 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:51:57.171 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:57.171 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:51:57.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T21:51:57.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:51:57.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:51:57.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T21:51:57.173 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:51:57.174 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:57.174 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:51:57.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T21:51:57.176 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:51:57.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:51:57.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:51:57.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T21:51:57.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T21:51:57.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T21:51:57.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:51:57.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:51:57.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:51:57.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T21:51:57.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T21:51:57.621 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:51:57.621 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:51:57.622 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:51:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:51:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:51:57.622 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:51:57.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:51:57.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:51:57.815 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:51:57.815 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:51:57.815 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:51:57.815 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:51:57.816 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:51:57.816 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:51:57.816 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:51:57.816 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:51:57.816 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:51:57.816 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:57.816 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:51:57.816 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T21:51:57.817 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:51:57.817 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T21:51:57.933 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:51:57.933 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:51:57.933 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:51:57.933 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:51:57.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:51:57.934 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:51:57.934 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:51:57.934 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:51:57.935 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:51:57.935 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:57.935 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:51:57.935 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T21:51:57.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:51:57.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T21:51:58.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T21:51:58.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T21:51:58.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:51:58.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:51:58.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:51:58.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T21:51:58.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:51:58.938 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:51:58.938 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:51:58.938 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:51:58.939 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:51:58.939 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:58.939 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:51:58.939 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:51:58.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:51:58.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:51:59.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T21:51:59.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:51:59.294 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T21:51:59.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:52:00.313 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T21:52:00.313 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:52:00.313 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:52:00.314 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:52:00.314 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:52:00.314 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:52:00.314 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T21:52:00.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T21:52:00.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T21:52:00.414 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:52:00.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T21:52:00.428 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:51:57.663+0000 7fa0c50a4e00 0 load: jerasure load: lrc 2024-09-15T21:52:00.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2024-09-15T21:52:00.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:379: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2024-09-15T21:52:00.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:380: TEST_ec_object_attr_read_error: local objname=myobject 2024-09-15T21:52:00.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:382: TEST_ec_object_attr_read_error: setup_osds 7 2024-09-15T21:52:00.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-15T21:52:00.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T21:52:00.436 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-15T21:52:00.436 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-15T21:52:00.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:52:00.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:52:00.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:52:00.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:52:00.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:52:00.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:52:00.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:52:00.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:52:00.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:52:00.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:52:00.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:52:00.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T21:52:00.439 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:52:00.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f057f331-fa31-4afb-b9ce-a3466bd32356 2024-09-15T21:52:00.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f057f331-fa31-4afb-b9ce-a3466bd32356' 2024-09-15T21:52:00.440 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 f057f331-fa31-4afb-b9ce-a3466bd32356 2024-09-15T21:52:00.440 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:52:00.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAAV+dmk70IGxAAOfukS7LM3kHjCMOnIH57lQ== 2024-09-15T21:52:00.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAAV+dmk70IGxAAOfukS7LM3kHjCMOnIH57lQ=="}' 2024-09-15T21:52:00.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f057f331-fa31-4afb-b9ce-a3466bd32356 -i td/test-erasure-eio/0/new.json 2024-09-15T21:52:00.753 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:52:00.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T21:52:00.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAAV+dmk70IGxAAOfukS7LM3kHjCMOnIH57lQ== --osd-uuid f057f331-fa31-4afb-b9ce-a3466bd32356 2024-09-15T21:52:00.798 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:00.795+0000 7fe4b0d5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:00.822 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:00.819+0000 7fe4b0d5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:00.824 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:00.823+0000 7fe4b0d5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:00.824 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:00.823+0000 7fe4b0d5a7c0 -1 bdev(0x55ed0e505c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:52:00.824 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:00.823+0000 7fe4b0d5a7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T21:52:06.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T21:52:06.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:52:06.816 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:52:06.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:52:06.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:52:07.380 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:52:07.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:52:07.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:52:07.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:52:07.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:52:07.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:52:07.414 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:07.411+0000 7f6865cd57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:07.435 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:07.435+0000 7f6865cd57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:07.436 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:07.435+0000 7f6865cd57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:07.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:52:07.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:52:07.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:52:07.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:52:07.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:52:07.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:07.863 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:52:07.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:52:07.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:07.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:08.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:08.721 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:08.719+0000 7f6865cd57c0 -1 Falling back to public interface 2024-09-15T21:52:09.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:09.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:09.350 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:52:09.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:52:09.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:09.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:09.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:10.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:10.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:10.787 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:52:10.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:52:10.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:10.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:11.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:11.532 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:11.531+0000 7f6865cd57c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:52:12.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:12.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:12.222 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:52:12.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:52:12.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:12.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:12.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:13.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:13.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:13.773 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:52:13.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:52:13.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:13.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:14.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:15.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:15.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:15.325 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:52:15.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:52:15.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:15.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:15.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:16.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:16.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:16.786 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:52:16.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:52:16.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:16.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:17.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:18.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:18.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:18.327 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:52:18.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:52:18.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:18.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:18.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:19.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:19.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:19.777 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:52:19.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:52:19.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:19.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:20.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:21.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:21.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:21.300 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:52:21.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:52:21.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:21.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:21.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:22.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:22.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:22.740 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:52:22.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:52:22.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:22.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:23.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:24.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:24.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:24.200 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:52:24.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:52:24.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:24.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:24.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:25.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:25.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:25.997 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:52:25.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:52:25.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:25.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:26.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:27.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:27.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:27.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:52:27.494 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:52:27.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:27.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:28.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:29.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:29.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:29.448 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:52:29.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:52:29.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:29.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:30.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:31.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:31.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:31.378 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:52:31.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:52:31.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:31.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:32.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:33.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:33.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:33.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:52:33.185 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:52:33.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:33.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:33.678 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3434693362,v1:127.0.0.1:6803/3434693362] [v2:127.0.0.1:6804/3434693362,v1:127.0.0.1:6805/3434693362] exists,up f057f331-fa31-4afb-b9ce-a3466bd32356 2024-09-15T21:52:33.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:52:33.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:52:33.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:52:33.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:52:33.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T21:52:33.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:52:33.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:52:33.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:52:33.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:52:33.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T21:52:33.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:52:33.683 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:52:33.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:52:33.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T21:52:33.685 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:52:33.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a49b4c57-e36a-4f82-975d-806d6efe293e 2024-09-15T21:52:33.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a49b4c57-e36a-4f82-975d-806d6efe293e' 2024-09-15T21:52:33.685 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 a49b4c57-e36a-4f82-975d-806d6efe293e 2024-09-15T21:52:33.686 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:52:33.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAhV+dmhPj9KRAA3kPWd/87XYh1ZK3MBIuB4Q== 2024-09-15T21:52:33.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAhV+dmhPj9KRAA3kPWd/87XYh1ZK3MBIuB4Q=="}' 2024-09-15T21:52:33.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a49b4c57-e36a-4f82-975d-806d6efe293e -i td/test-erasure-eio/1/new.json 2024-09-15T21:52:34.279 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:52:34.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T21:52:34.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhV+dmhPj9KRAA3kPWd/87XYh1ZK3MBIuB4Q== --osd-uuid a49b4c57-e36a-4f82-975d-806d6efe293e 2024-09-15T21:52:34.330 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:34.327+0000 7f3dc762b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:34.339 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:34.339+0000 7f3dc762b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:34.341 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:34.339+0000 7f3dc762b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:34.341 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:34.339+0000 7f3dc762b7c0 -1 bdev(0x563146383c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:52:34.341 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:34.339+0000 7f3dc762b7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T21:52:39.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T21:52:39.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:52:39.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:52:39.938 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T21:52:39.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:52:40.537 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T21:52:40.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:52:40.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:52:40.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:52:40.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:52:40.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:52:40.569 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:40.567+0000 7fd6dd5a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:40.594 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:40.591+0000 7fd6dd5a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:40.595 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:40.595+0000 7fd6dd5a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:41.025 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:52:41.025 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:52:41.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:52:41.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:52:41.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:52:41.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:41.026 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:52:41.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:52:41.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:41.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:41.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:41.877 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:41.875+0000 7fd6dd5a17c0 -1 Falling back to public interface 2024-09-15T21:52:42.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:42.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:42.503 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:52:42.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:52:42.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:42.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:42.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:43.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:43.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:43.934 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:52:43.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:52:43.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:43.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:44.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:44.814 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:52:44.811+0000 7fd6dd5a17c0 -1 osd.1 0 log_to_monitors true 2024-09-15T21:52:45.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:45.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:45.466 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:52:45.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:52:45.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:45.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:46.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:47.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:47.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:47.038 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:52:47.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:52:47.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:47.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:47.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:48.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:48.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:48.582 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:52:48.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:52:48.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:48.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:49.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:50.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:50.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:50.013 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:52:50.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:52:50.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:50.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:50.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:51.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:51.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:51.465 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:52:51.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:52:51.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:51.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:51.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:52.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:52.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:52.916 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:52:52.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:52:52.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:52.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:53.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:54.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:54.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:54.364 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:52:54.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:52:54.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:54.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:54.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:55.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:55.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:55.796 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:52:55.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:52:55.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:55.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:56.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:57.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:57.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:57.369 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:52:57.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:52:57.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:57.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:58.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:59.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:59.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:59.263 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:52:59.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:52:59.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:59.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:53:00.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:01.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:01.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:01.452 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:53:01.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:53:01.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:01.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:53:02.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:03.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:03.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:03.173 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:53:03.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:53:03.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:03.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:53:03.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:04.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:04.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:04.976 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:53:04.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:53:04.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:04.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:53:05.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:06.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:06.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:06.714 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:53:06.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:53:06.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:06.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:53:07.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:08.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:08.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:08.508 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:53:08.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:53:08.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:08.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:53:08.973 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/398704036,v1:127.0.0.1:6811/398704036] [v2:127.0.0.1:6812/398704036,v1:127.0.0.1:6813/398704036] exists,up a49b4c57-e36a-4f82-975d-806d6efe293e 2024-09-15T21:53:08.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:53:08.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:53:08.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:53:08.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:53:08.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T21:53:08.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:53:08.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:53:08.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:53:08.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:53:08.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T21:53:08.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:53:08.977 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:53:08.978 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:53:08.978 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:53:08.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:53:08.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:53:08.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:53:08.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:53:08.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:53:08.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:53:08.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:53:08.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:53:08.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:53:08.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:53:08.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:53:08.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:53:08.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T21:53:08.980 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:53:08.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b389fc98-7ad0-482e-8de3-62638175dd85 2024-09-15T21:53:08.980 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 b389fc98-7ad0-482e-8de3-62638175dd85 2024-09-15T21:53:08.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 b389fc98-7ad0-482e-8de3-62638175dd85' 2024-09-15T21:53:08.981 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:53:08.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBEV+dmdOlwOxAA0ul6KvY/iF7xXiUGDmGCFA== 2024-09-15T21:53:08.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBEV+dmdOlwOxAA0ul6KvY/iF7xXiUGDmGCFA=="}' 2024-09-15T21:53:08.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b389fc98-7ad0-482e-8de3-62638175dd85 -i td/test-erasure-eio/2/new.json 2024-09-15T21:53:09.497 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:53:09.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T21:53:09.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEV+dmdOlwOxAA0ul6KvY/iF7xXiUGDmGCFA== --osd-uuid b389fc98-7ad0-482e-8de3-62638175dd85 2024-09-15T21:53:09.547 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:09.547+0000 7f4c186207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:09.550 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:09.547+0000 7f4c186207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:09.551 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:09.551+0000 7f4c186207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:09.551 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:09.551+0000 7f4c186207c0 -1 bdev(0x55c7f16dbc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:53:09.551 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:09.551+0000 7f4c186207c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T21:53:15.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T21:53:15.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:53:15.432 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T21:53:15.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:53:15.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:53:16.012 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T21:53:16.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:53:16.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:53:16.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:53:16.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:53:16.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:53:16.038 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:16.034+0000 7fdb3944c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:16.054 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:16.050+0000 7fdb3944c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:16.055 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:16.054+0000 7fdb3944c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:16.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:53:16.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:53:16.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:53:16.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:53:16.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:53:16.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:16.475 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:53:16.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:53:16.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:16.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:16.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:17.533 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:17.530+0000 7fdb3944c7c0 -1 Falling back to public interface 2024-09-15T21:53:17.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:17.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:17.927 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:53:17.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:53:17.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:17.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:18.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:19.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:19.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:19.348 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:53:19.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:53:19.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:19.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:19.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:20.480 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:20.478+0000 7fdb3944c7c0 -1 osd.2 0 log_to_monitors true 2024-09-15T21:53:20.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:20.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:20.801 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:53:20.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:53:20.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:20.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:21.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:22.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:22.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:22.276 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:53:22.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:53:22.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:22.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:22.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:23.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:23.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:23.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:53:23.780 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:53:23.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:23.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:24.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:25.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:25.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:25.269 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:53:25.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:53:25.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:25.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:25.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:26.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:26.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:26.703 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:53:26.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:53:26.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:26.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:27.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:28.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:28.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:28.225 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:53:28.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:53:28.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:28.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:28.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:29.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:29.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:29.667 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:53:29.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:53:29.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:29.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:30.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:31.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:31.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:31.235 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:53:31.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:53:31.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:31.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:31.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:32.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:32.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:32.690 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:53:32.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:53:32.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:32.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:33.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:34.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:34.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:34.226 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:53:34.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:53:34.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:34.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:34.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:35.675 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:53:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:53:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:37.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:37.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:37.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:53:37.256 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:53:37.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:37.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:37.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:38.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:38.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:38.920 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:53:38.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:53:38.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:38.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:39.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:40.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:40.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:40.655 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:53:40.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:53:40.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:40.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:41.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:42.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:42.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:42.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:53:42.221 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:53:42.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:42.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:42.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:43.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:43.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:43.643 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:53:43.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:53:43.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:43.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:44.276 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3832879421,v1:127.0.0.1:6819/3832879421] [v2:127.0.0.1:6820/3832879421,v1:127.0.0.1:6821/3832879421] exists,up b389fc98-7ad0-482e-8de3-62638175dd85 2024-09-15T21:53:44.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:53:44.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:53:44.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:53:44.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T21:53:44.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T21:53:44.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:53:44.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:53:44.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:53:44.280 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:53:44.280 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:53:44.280 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:53:44.282 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:53:44.282 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:53:44.282 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:53:44.282 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:53:44.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:53:44.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:53:44.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:53:44.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:53:44.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:53:44.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:53:44.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:53:44.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:53:44.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:53:44.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:53:44.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:53:44.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:53:44.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T21:53:44.283 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:53:44.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=74eb1623-abd3-46a5-baca-013c56db32ed 2024-09-15T21:53:44.284 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 74eb1623-abd3-46a5-baca-013c56db32ed 2024-09-15T21:53:44.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 74eb1623-abd3-46a5-baca-013c56db32ed' 2024-09-15T21:53:44.284 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:53:44.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBoV+dm7bFHEhAAUtJpi2CzCQr8T9tHDMIoyg== 2024-09-15T21:53:44.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBoV+dm7bFHEhAAUtJpi2CzCQr8T9tHDMIoyg=="}' 2024-09-15T21:53:44.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 74eb1623-abd3-46a5-baca-013c56db32ed -i td/test-erasure-eio/3/new.json 2024-09-15T21:53:45.107 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:53:45.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T21:53:45.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBoV+dm7bFHEhAAUtJpi2CzCQr8T9tHDMIoyg== --osd-uuid 74eb1623-abd3-46a5-baca-013c56db32ed 2024-09-15T21:53:45.157 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:45.154+0000 7fc1aa2eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:45.159 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:45.158+0000 7fc1aa2eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:45.160 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:45.158+0000 7fc1aa2eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:45.160 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:45.158+0000 7fc1aa2eb7c0 -1 bdev(0x5582557cfc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T21:53:45.161 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:45.158+0000 7fc1aa2eb7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T21:53:51.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T21:53:51.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:53:51.566 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T21:53:51.566 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T21:53:51.566 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:53:52.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T21:53:52.142 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T21:53:52.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:53:52.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:53:52.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:53:52.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:53:52.168 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:52.166+0000 7f2778c057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:52.170 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:52.170+0000 7f2778c057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:52.172 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:52.170+0000 7f2778c057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:52.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T21:53:52.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:53:52.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:53:52.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:53:52.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:53:52.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:52.607 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:53:52.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:53:52.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:52.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:53.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:53.361 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:53.358+0000 7f2778c057c0 -1 Falling back to public interface 2024-09-15T21:53:54.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:54.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:54.092 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:53:54.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:53:54.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:54.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:54.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:55.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:55.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:55.520 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:53:55.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:53:55.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:55.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:55.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:56.328 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:53:56.326+0000 7f2778c057c0 -1 osd.3 0 log_to_monitors true 2024-09-15T21:53:56.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:56.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:56.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:53:56.983 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:53:56.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:56.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:57.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:58.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:58.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:58.713 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:53:58.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:53:58.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:58.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:59.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:00.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:00.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:00.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:54:00.179 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:54:00.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:00.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:00.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:01.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:01.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:01.651 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:54:01.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:54:01.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:01.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:02.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:03.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:03.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:03.156 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:54:03.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:54:03.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:03.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:03.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:04.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:04.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:04.610 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:54:04.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:54:04.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:04.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:05.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:06.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:06.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:06.171 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:54:06.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:54:06.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:06.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:06.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:07.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:07.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:07.621 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:54:07.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:54:07.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:07.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:08.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:09.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:09.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:09.156 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:54:09.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:54:09.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:09.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:09.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:10.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:10.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:10.676 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:54:10.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:54:10.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:10.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:11.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:12.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:12.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:12.458 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:54:12.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:54:12.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:12.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:13.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:14.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:14.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:14.427 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:54:14.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:54:14.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:14.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:15.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:16.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:16.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:16.496 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:54:16.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:54:16.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:16.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:17.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:18.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:18.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:18.026 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:54:18.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:54:18.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:18.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:19.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:20.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:20.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:20.122 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:54:20.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:54:20.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:20.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:20.595 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2663224532,v1:127.0.0.1:6827/2663224532] [v2:127.0.0.1:6828/2663224532,v1:127.0.0.1:6829/2663224532] exists,up 74eb1623-abd3-46a5-baca-013c56db32ed 2024-09-15T21:54:20.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:54:20.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:54:20.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:54:20.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:54:20.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T21:54:20.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:54:20.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:54:20.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:54:20.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:54:20.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:54:20.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:54:20.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:54:20.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:54:20.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:54:20.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T21:54:20.601 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:54:20.602 INFO:tasks.workunit.client.0.smithi104.stdout:add osd4 9430ce4b-2b54-4c4c-97b5-1f2db072adeb 2024-09-15T21:54:20.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9430ce4b-2b54-4c4c-97b5-1f2db072adeb 2024-09-15T21:54:20.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 9430ce4b-2b54-4c4c-97b5-1f2db072adeb' 2024-09-15T21:54:20.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:54:20.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCMV+dmYVDlJBAAwqg/fX0+jG3cf/n9l+RenA== 2024-09-15T21:54:20.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCMV+dmYVDlJBAAwqg/fX0+jG3cf/n9l+RenA=="}' 2024-09-15T21:54:20.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9430ce4b-2b54-4c4c-97b5-1f2db072adeb -i td/test-erasure-eio/4/new.json 2024-09-15T21:54:21.408 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:54:21.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T21:54:21.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMV+dmYVDlJBAAwqg/fX0+jG3cf/n9l+RenA== --osd-uuid 9430ce4b-2b54-4c4c-97b5-1f2db072adeb 2024-09-15T21:54:21.468 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:21.466+0000 7fbb6210b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:21.472 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:21.470+0000 7fbb6210b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:21.473 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:21.470+0000 7fbb6210b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:21.474 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:21.470+0000 7fbb6210b7c0 -1 bdev(0x5623b3ddfc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-15T21:54:21.474 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:21.474+0000 7fbb6210b7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T21:54:28.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T21:54:28.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:54:28.074 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd4 key to auth repository 2024-09-15T21:54:28.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-15T21:54:28.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:54:28.651 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.4 2024-09-15T21:54:28.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-15T21:54:28.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:54:28.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:54:28.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:54:28.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:54:28.684 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:28.682+0000 7ff3c31877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:28.700 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:28.698+0000 7ff3c31877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:28.701 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:28.698+0000 7ff3c31877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:29.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-15T21:54:29.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:54:29.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-15T21:54:29.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:54:29.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:54:29.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:29.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:54:29.074 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:54:29.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:29.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:29.490 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:29.981 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:29.978+0000 7ff3c31877c0 -1 Falling back to public interface 2024-09-15T21:54:30.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:30.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:30.492 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:54:30.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:54:30.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:30.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:30.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:31.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:31.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:31.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:54:31.959 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:54:31.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:31.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:32.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:32.810 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:32.806+0000 7ff3c31877c0 -1 osd.4 0 log_to_monitors true 2024-09-15T21:54:33.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:33.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:33.390 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:54:33.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:54:33.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:33.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:34.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:35.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:35.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:35.114 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:54:35.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:54:35.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:35.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:36.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:36.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:36.811 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:54:36.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:54:36.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:36.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:37.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:38.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:38.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:38.256 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:54:38.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:54:38.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:38.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:38.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:39.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:39.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:39.771 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:54:39.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:54:39.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:39.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:40.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:41.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:41.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:41.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:54:41.348 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:54:41.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:41.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:41.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:42.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:42.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:42.919 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:54:42.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:54:42.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:42.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:43.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:44.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:44.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:44.339 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:54:44.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:54:44.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:44.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:44.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:45.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:45.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:45.811 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:54:45.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:54:45.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:45.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:46.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:47.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:47.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:47.282 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:54:47.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:54:47.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:47.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:49.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:49.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:49.304 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:54:49.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:54:49.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:49.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:50.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:51.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:51.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:51.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:54:51.073 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:54:51.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:51.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:51.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:52.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:52.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:52.832 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:54:52.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:54:52.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:52.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:53.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:54.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:54.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:54.686 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:54:54.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:54:54.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:54.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:55.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:56.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:56.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:56.369 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:54:56.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:54:56.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:56.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:54:56.988 INFO:tasks.workunit.client.0.smithi104.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/536842048,v1:127.0.0.1:6835/536842048] [v2:127.0.0.1:6836/536842048,v1:127.0.0.1:6837/536842048] exists,up 9430ce4b-2b54-4c4c-97b5-1f2db072adeb 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:54:56.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:54:56.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:54:56.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-15T21:54:56.995 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:54:56.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4d5cf99d-4735-484f-8fb6-569a0572d7f8 2024-09-15T21:54:56.995 INFO:tasks.workunit.client.0.smithi104.stdout:add osd5 4d5cf99d-4735-484f-8fb6-569a0572d7f8 2024-09-15T21:54:56.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 4d5cf99d-4735-484f-8fb6-569a0572d7f8' 2024-09-15T21:54:56.996 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:54:57.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCxV+dm0DLFABAARV98yxhEAIlVgwF6EiFQ5g== 2024-09-15T21:54:57.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCxV+dm0DLFABAARV98yxhEAIlVgwF6EiFQ5g=="}' 2024-09-15T21:54:57.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4d5cf99d-4735-484f-8fb6-569a0572d7f8 -i td/test-erasure-eio/5/new.json 2024-09-15T21:54:57.860 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:54:57.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-15T21:54:57.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxV+dm0DLFABAARV98yxhEAIlVgwF6EiFQ5g== --osd-uuid 4d5cf99d-4735-484f-8fb6-569a0572d7f8 2024-09-15T21:54:57.912 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:57.910+0000 7f82f1fe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:57.975 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:57.974+0000 7f82f1fe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:57.977 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:57.974+0000 7f82f1fe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:57.977 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:57.974+0000 7f82f1fe47c0 -1 bdev(0x5638e5ef3c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-15T21:54:57.977 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:54:57.974+0000 7f82f1fe47c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-15T21:55:04.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-15T21:55:04.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:55:04.658 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd5 key to auth repository 2024-09-15T21:55:04.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-15T21:55:04.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:55:05.224 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.5 2024-09-15T21:55:05.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-15T21:55:05.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:55:05.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:55:05.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:55:05.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:55:05.255 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:05.253+0000 7f86c85537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:05.266 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:05.261+0000 7f86c85537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:05.267 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:05.265+0000 7f86c85537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:05.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-15T21:55:05.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:55:05.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:55:05.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:55:05.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:55:05.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:05.638 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:55:05.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:55:05.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:05.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:06.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:06.809 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:06.805+0000 7f86c85537c0 -1 Falling back to public interface 2024-09-15T21:55:07.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:07.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:07.064 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:55:07.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:55:07.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:07.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:07.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:08.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:08.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:08.524 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:55:08.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:55:08.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:08.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:08.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:09.709 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:09.705+0000 7f86c85537c0 -1 osd.5 0 log_to_monitors true 2024-09-15T21:55:09.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:09.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:09.990 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:55:09.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:55:09.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:09.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:10.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:11.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:11.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:11.547 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:55:11.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:55:11.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:11.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:12.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:13.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:13.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:13.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:55:13.204 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:55:13.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:13.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:13.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:14.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:14.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:14.696 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:55:14.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:55:14.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:14.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:15.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:16.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:16.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:16.150 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:55:16.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:55:16.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:16.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:16.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:17.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:17.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:17.802 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:55:17.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:55:17.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:17.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:18.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:19.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:19.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:19.357 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:55:19.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:55:19.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:19.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:19.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:20.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:20.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:20.826 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:55:20.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:55:20.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:20.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:21.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:22.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:22.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:22.283 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:55:22.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:55:22.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:22.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:23.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:24.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:24.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:24.253 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:55:24.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:55:24.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:24.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:25.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:26.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:26.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:26.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:55:26.637 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:55:26.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:26.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:27.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:28.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:28.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:28.641 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:55:28.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:55:28.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:28.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:29.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:30.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:30.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:30.648 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:55:30.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:55:30.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:30.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:31.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:32.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:32.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:32.438 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:55:32.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:55:32.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:32.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:32.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:33.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:33.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:33.920 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:55:33.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:55:33.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:33.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:55:34.546 INFO:tasks.workunit.client.0.smithi104.stdout:osd.5 up in weight 1 up_from 47 up_thru 49 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/778773674,v1:127.0.0.1:6843/778773674] [v2:127.0.0.1:6844/778773674,v1:127.0.0.1:6845/778773674] exists,up 4d5cf99d-4735-484f-8fb6-569a0572d7f8 2024-09-15T21:55:34.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:55:34.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:55:34.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:55:34.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:55:34.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-15T21:55:34.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:55:34.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:55:34.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-15T21:55:34.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:55:34.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-15T21:55:34.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:55:34.549 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:55:34.550 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:55:34.550 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:55:34.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:55:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-15T21:55:34.576 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:55:34.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=aa78d1ee-cc1b-4ccf-b1a8-5fca7fc9459d 2024-09-15T21:55:34.578 INFO:tasks.workunit.client.0.smithi104.stdout:add osd6 aa78d1ee-cc1b-4ccf-b1a8-5fca7fc9459d 2024-09-15T21:55:34.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 aa78d1ee-cc1b-4ccf-b1a8-5fca7fc9459d' 2024-09-15T21:55:34.579 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:55:34.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDWV+dmKtHeIxAAWYR/+EOJobezYQNrvL+O2w== 2024-09-15T21:55:34.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDWV+dmKtHeIxAAWYR/+EOJobezYQNrvL+O2w=="}' 2024-09-15T21:55:34.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new aa78d1ee-cc1b-4ccf-b1a8-5fca7fc9459d -i td/test-erasure-eio/6/new.json 2024-09-15T21:55:35.136 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:55:35.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-15T21:55:35.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWV+dmKtHeIxAAWYR/+EOJobezYQNrvL+O2w== --osd-uuid aa78d1ee-cc1b-4ccf-b1a8-5fca7fc9459d 2024-09-15T21:55:35.187 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:35.185+0000 7f644007e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:35.278 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:35.277+0000 7f644007e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:35.280 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:35.277+0000 7f644007e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:35.281 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:35.277+0000 7f644007e7c0 -1 bdev(0x55f22c05bc00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-15T21:55:35.281 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:35.277+0000 7f644007e7c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-15T21:55:41.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-15T21:55:41.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:55:41.196 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd6 key to auth repository 2024-09-15T21:55:41.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-15T21:55:41.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:55:41.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-15T21:55:41.769 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.6 2024-09-15T21:55:41.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:55:41.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:55:41.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:55:41.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:55:41.800 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:41.797+0000 7f2a2bb8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:41.806 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:41.805+0000 7f2a2bb8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:41.808 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:41.805+0000 7f2a2bb8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:42.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-15T21:55:42.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:55:42.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-15T21:55:42.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:55:42.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:55:42.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:42.203 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:55:42.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:55:42.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:42.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:42.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:43.309 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:43.305+0000 7f2a2bb8d7c0 -1 Falling back to public interface 2024-09-15T21:55:43.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:43.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:43.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:55:43.682 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:55:43.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:43.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:44.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:45.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:45.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:45.117 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:55:45.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:55:45.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:45.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:45.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:46.156 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:55:46.153+0000 7f2a2bb8d7c0 -1 osd.6 0 log_to_monitors true 2024-09-15T21:55:46.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:46.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:46.536 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:55:46.537 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:55:46.537 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:46.537 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:47.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:48.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:48.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:48.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:55:48.241 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:55:48.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:48.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:48.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:49.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:49.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:49.864 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:55:49.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:55:49.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:49.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:50.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:51.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:51.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:51.353 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:55:51.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:55:51.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:51.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:51.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:52.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:52.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:52.829 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:55:52.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:55:52.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:52.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:53.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:54.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:55.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:55.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:55.926 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:55:55.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:55:55.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:55.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:56.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:57.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:57.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:57.477 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:55:57.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:55:57.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:57.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:57.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:58.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:58.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:58.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:55:58.964 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:55:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:58.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:55:59.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:00.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:00.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:00.842 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:56:00.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:56:00.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:00.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:56:01.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:02.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:02.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:02.526 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:56:02.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:56:02.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:02.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:56:03.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:04.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:04.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:04.390 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:56:04.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:56:04.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:04.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:56:05.571 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:06.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:06.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:06.573 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:56:06.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:56:06.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:06.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:56:07.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:08.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:08.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:08.398 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:56:08.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:56:08.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:08.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:56:09.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:10.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:10.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:10.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:56:10.323 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:56:10.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:10.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:56:10.772 INFO:tasks.workunit.client.0.smithi104.stdout:osd.6 up in weight 1 up_from 57 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1858099080,v1:127.0.0.1:6851/1858099080] [v2:127.0.0.1:6852/1858099080,v1:127.0.0.1:6853/1858099080] exists,up aa78d1ee-cc1b-4ccf-b1a8-5fca7fc9459d 2024-09-15T21:56:10.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:56:10.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:56:10.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:56:10.773 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T21:56:10.773 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:56:10.773 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:56:10.774 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:56:10.774 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:10.774 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:10.774 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T21:56:10.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T21:56:10.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T21:56:10.884 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:56:10.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T21:56:10.902 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:52:08.731+0000 7f6865cd57c0 0 load: jerasure load: lrc 2024-09-15T21:56:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:384: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2024-09-15T21:56:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T21:56:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T21:56:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T21:56:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T21:56:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T21:56:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T21:56:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T21:56:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T21:56:11.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T21:56:11.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T21:56:12.916 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T21:56:12.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:56:13.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T21:56:13.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:56:13.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:56:13.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:56:13.939 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:56:13.940 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:56:13.940 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:56:13.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:56:13.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:56:13.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:56:14.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:56:14.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:56:14.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:56:14.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:56:14.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:56:14.067 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:56:14.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:56:14.818 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:56:14.818 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:56:14.818 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:56:14.819 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:56:14.819 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:56:14.819 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T21:56:14.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:56:14.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:14.820 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:56:14.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803831 2024-09-15T21:56:14.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803831 2024-09-15T21:56:14.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831' 2024-09-15T21:56:14.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:14.986 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:56:15.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607598 2024-09-15T21:56:15.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607598 2024-09-15T21:56:15.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-51539607598' 2024-09-15T21:56:15.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:15.138 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:56:15.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411369 2024-09-15T21:56:15.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411369 2024-09-15T21:56:15.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-51539607598 2-77309411369' 2024-09-15T21:56:15.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:15.291 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:56:15.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084319 2024-09-15T21:56:15.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084319 2024-09-15T21:56:15.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-51539607598 2-77309411369 3-120259084319' 2024-09-15T21:56:15.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:15.449 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:56:15.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757270 2024-09-15T21:56:15.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757270 2024-09-15T21:56:15.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-51539607598 2-77309411369 3-120259084319 4-163208757270' 2024-09-15T21:56:15.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:15.601 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:56:15.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462928 2024-09-15T21:56:15.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462928 2024-09-15T21:56:15.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-51539607598 2-77309411369 3-120259084319 4-163208757270 5-201863462928' 2024-09-15T21:56:15.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:15.755 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:56:15.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135880 2024-09-15T21:56:15.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135880 2024-09-15T21:56:15.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-51539607598 2-77309411369 3-120259084319 4-163208757270 5-201863462928 6-244813135880' 2024-09-15T21:56:15.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:15.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803831 2024-09-15T21:56:15.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:15.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:56:15.905 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803831 2024-09-15T21:56:15.905 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:15.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803831 2024-09-15T21:56:15.907 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803831 2024-09-15T21:56:15.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-09-15T21:56:15.907 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:56:16.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803831 -lt 25769803831 2024-09-15T21:56:16.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:16.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607598 2024-09-15T21:56:16.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:16.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:56:16.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607598 2024-09-15T21:56:16.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:16.332 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 51539607598 2024-09-15T21:56:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607598 2024-09-15T21:56:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607598' 2024-09-15T21:56:16.332 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:56:16.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607598 -lt 51539607598 2024-09-15T21:56:16.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:16.754 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411369 2024-09-15T21:56:16.755 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:16.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:56:16.757 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411369 2024-09-15T21:56:16.757 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:16.758 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 77309411369 2024-09-15T21:56:16.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411369 2024-09-15T21:56:16.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411369' 2024-09-15T21:56:16.758 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:56:17.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411369 -lt 77309411369 2024-09-15T21:56:17.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:17.168 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:17.169 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084319 2024-09-15T21:56:17.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:56:17.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084319 2024-09-15T21:56:17.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:17.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084319 2024-09-15T21:56:17.171 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 120259084319 2024-09-15T21:56:17.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084319' 2024-09-15T21:56:17.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:56:17.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084319 -lt 120259084319 2024-09-15T21:56:17.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:17.594 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757270 2024-09-15T21:56:17.595 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:17.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:56:17.597 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757270 2024-09-15T21:56:17.597 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:17.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757270 2024-09-15T21:56:17.599 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 163208757270 2024-09-15T21:56:17.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757270' 2024-09-15T21:56:17.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:56:18.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757271 -lt 163208757270 2024-09-15T21:56:18.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:18.044 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:18.045 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-201863462928 2024-09-15T21:56:18.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:56:18.046 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-201863462928 2024-09-15T21:56:18.046 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:18.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462928 2024-09-15T21:56:18.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 201863462928' 2024-09-15T21:56:18.048 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 201863462928 2024-09-15T21:56:18.048 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:56:18.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462928 -lt 201863462928 2024-09-15T21:56:18.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:18.548 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-244813135880 2024-09-15T21:56:18.549 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:18.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:56:18.551 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-244813135880 2024-09-15T21:56:18.551 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:18.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135880 2024-09-15T21:56:18.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 244813135880' 2024-09-15T21:56:18.553 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 244813135880 2024-09-15T21:56:18.553 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:56:18.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135880 -lt 244813135880 2024-09-15T21:56:18.992 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:56:18.993 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:56:18.993 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:56:19.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:56:19.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:56:19.554 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:56:19.554 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:56:19.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:56:19.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:56:19.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:56:19.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:56:19.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:56:19.957 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:56:19.957 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:56:19.957 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:56:20.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:56:20.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:56:20.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:56:20.498 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:387: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2024-09-15T21:56:20.498 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=pool-jerasure 2024-09-15T21:56:20.498 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=myobject 2024-09-15T21:56:20.498 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-15T21:56:20.498 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:56:20.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:387: TEST_ec_object_attr_read_error: local primary_osd=3 2024-09-15T21:56:20.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:389: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-15T21:56:20.963 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:56:20.963 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:56:20.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:56:20.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:56:20.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:56:21.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:56:21.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:392: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-15T21:56:21.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T21:56:21.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T21:56:21.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-15T21:56:21.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T21:56:21.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T21:56:21.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T21:56:21.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T21:56:21.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T21:56:21.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T21:56:21.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T21:56:21.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T21:56:21.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-15T21:56:22.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2024-09-15T21:56:22.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T21:56:22.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T21:56:22.507 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-15T21:56:22.508 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T21:56:22.508 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:56:22.993 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2147483647 2024-09-15T21:56:22.993 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:56:22.994 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:56:22.994 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T21:56:22.994 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T21:56:22.994 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2147483647 5 1 0 6 2024-09-15T21:56:22.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('2147483647' '5' '1' '0' '6') 2024-09-15T21:56:22.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T21:56:22.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=5 2024-09-15T21:56:22.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T21:56:22.994 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/5/type 2024-09-15T21:56:22.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T21:56:22.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2024-09-15T21:56:22.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T21:56:22.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=5 2024-09-15T21:56:22.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T21:56:22.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T21:56:22.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T21:56:22.997 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.5 2024-09-15T21:56:22.997 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.5 2024-09-15T21:56:22.997 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.5 ']' 2024-09-15T21:56:22.997 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:56:22.997 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:22.997 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:22.997 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.5.asok 2024-09-15T21:56:22.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2024-09-15T21:56:23.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T21:56:23.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T21:56:23.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T21:56:23.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.5 2024-09-15T21:56:23.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.5 2024-09-15T21:56:23.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.5 ']' 2024-09-15T21:56:23.114 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:56:23.114 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:23.114 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:23.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.5.asok 2024-09-15T21:56:23.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T21:56:23.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2024-09-15T21:56:23.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2024-09-15T21:56:23.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T21:56:23.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:56:23.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T21:56:23.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:56:23.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:56:23.243 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:23.244 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:23.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:56:23.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:56:23.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:56:23.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:56:23.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:56:23.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:56:23.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:56:23.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:56:23.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:56:23.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:56:23.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:56:23.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T21:56:23.246 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T21:56:23.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T21:56:23.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:56:23.247 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-15T21:56:23.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T21:56:23.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:56:23.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:56:23.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:56:23.277 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:23.273+0000 7f9401a917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:23.280 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:23.277+0000 7f9401a917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:23.281 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:23.277+0000 7f9401a917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:23.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T21:56:23.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:56:23.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:56:23.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:56:23.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:56:23.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:23.662 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:56:23.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:56:23.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:23.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:24.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:24.513 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:24.509+0000 7f9401a917c0 -1 Falling back to public interface 2024-09-15T21:56:25.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:25.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:25.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:56:25.100 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:56:25.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:25.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:25.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:26.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:26.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:26.579 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:56:26.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:56:26.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:26.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:26.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:27.754 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:27.749+0000 7f9401a917c0 -1 osd.3 65 log_to_monitors true 2024-09-15T21:56:27.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:27.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:27.999 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:56:27.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:56:27.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:27.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:28.679 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 69 up_thru 69 down_at 66 last_clean_interval [28,65) [v2:127.0.0.1:6826/441107259,v1:127.0.0.1:6827/441107259] [v2:127.0.0.1:6828/441107259,v1:127.0.0.1:6829/441107259] exists,up 74eb1623-abd3-46a5-baca-013c56db32ed 2024-09-15T21:56:28.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:56:28.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:56:28.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:56:28.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_object_attr_read_error: wait_for_clean 2024-09-15T21:56:28.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:56:28.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:56:28.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:56:28.680 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:56:28.681 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:56:28.681 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:56:28.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:56:28.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:56:28.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:56:28.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:56:28.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:56:28.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:56:28.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:56:28.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:56:28.789 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:56:29.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:56:29.216 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T21:56:29.216 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T21:56:29.217 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T21:56:29.217 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T21:56:29.217 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T21:56:29.217 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T21:56:29.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:56:29.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:29.217 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:56:29.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803836 2024-09-15T21:56:29.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803836 2024-09-15T21:56:29.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836' 2024-09-15T21:56:29.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:29.377 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:56:29.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607603 2024-09-15T21:56:29.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607603 2024-09-15T21:56:29.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-51539607603' 2024-09-15T21:56:29.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:29.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:56:29.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411374 2024-09-15T21:56:29.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411374 2024-09-15T21:56:29.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-51539607603 2-77309411374' 2024-09-15T21:56:29.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:29.659 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:56:29.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=296352743427 2024-09-15T21:56:29.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 296352743427 2024-09-15T21:56:29.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-51539607603 2-77309411374 3-296352743427' 2024-09-15T21:56:29.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:29.819 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:56:29.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757275 2024-09-15T21:56:29.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757275 2024-09-15T21:56:29.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-51539607603 2-77309411374 3-296352743427 4-163208757275' 2024-09-15T21:56:29.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:29.965 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:56:30.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462933 2024-09-15T21:56:30.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462933 2024-09-15T21:56:30.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-51539607603 2-77309411374 3-296352743427 4-163208757275 5-201863462933' 2024-09-15T21:56:30.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:56:30.115 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:56:30.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135884 2024-09-15T21:56:30.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135884 2024-09-15T21:56:30.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-51539607603 2-77309411374 3-296352743427 4-163208757275 5-201863462933 6-244813135884' 2024-09-15T21:56:30.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:30.266 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803836 2024-09-15T21:56:30.266 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:30.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:56:30.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803836 2024-09-15T21:56:30.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:30.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803836 2024-09-15T21:56:30.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803836' 2024-09-15T21:56:30.269 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803836 2024-09-15T21:56:30.270 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:56:30.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803836 -lt 25769803836 2024-09-15T21:56:30.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:30.678 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607603 2024-09-15T21:56:30.678 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:30.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:56:30.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607603 2024-09-15T21:56:30.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:30.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607603 2024-09-15T21:56:30.683 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 51539607603 2024-09-15T21:56:30.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607603' 2024-09-15T21:56:30.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:56:31.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607603 -lt 51539607603 2024-09-15T21:56:31.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:31.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411374 2024-09-15T21:56:31.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:31.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:56:31.144 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411374 2024-09-15T21:56:31.145 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:31.145 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 77309411374 2024-09-15T21:56:31.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411374 2024-09-15T21:56:31.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411374' 2024-09-15T21:56:31.146 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:56:31.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411374 -lt 77309411374 2024-09-15T21:56:31.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:31.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:31.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-296352743427 2024-09-15T21:56:31.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:56:31.604 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:31.604 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-296352743427 2024-09-15T21:56:31.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=296352743427 2024-09-15T21:56:31.605 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 296352743427 2024-09-15T21:56:31.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 296352743427' 2024-09-15T21:56:31.606 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:56:32.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 296352743427 -lt 296352743427 2024-09-15T21:56:32.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:32.162 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757275 2024-09-15T21:56:32.162 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:32.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:56:32.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:32.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757275 2024-09-15T21:56:32.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757275 2024-09-15T21:56:32.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757275' 2024-09-15T21:56:32.166 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 163208757275 2024-09-15T21:56:32.166 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:56:32.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757275 -lt 163208757275 2024-09-15T21:56:32.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:32.698 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-201863462933 2024-09-15T21:56:32.698 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:32.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:56:32.700 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-201863462933 2024-09-15T21:56:32.700 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:32.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462933 2024-09-15T21:56:32.702 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 201863462933 2024-09-15T21:56:32.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 201863462933' 2024-09-15T21:56:32.702 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:56:33.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462933 -lt 201863462933 2024-09-15T21:56:33.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:56:33.222 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:56:33.222 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-244813135884 2024-09-15T21:56:33.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:56:33.224 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-244813135884 2024-09-15T21:56:33.224 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:56:33.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135884 2024-09-15T21:56:33.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 244813135884' 2024-09-15T21:56:33.226 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 244813135884 2024-09-15T21:56:33.226 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:56:33.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135885 -lt 244813135884 2024-09-15T21:56:33.676 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:56:33.676 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:56:33.677 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:56:34.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:56:34.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:56:34.234 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:56:34.234 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:56:34.234 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:56:34.234 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:56:34.235 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:56:34.235 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:56:34.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:56:34.669 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:56:34.670 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:56:34.670 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:56:35.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:56:35.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:56:35.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:56:35.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:403: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-15T21:56:35.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T21:56:35.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T21:56:35.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-15T21:56:35.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T21:56:35.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T21:56:35.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-15T21:56:35.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T21:56:35.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T21:56:35.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2024-09-15T21:56:35.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T21:56:35.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T21:56:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T21:56:35.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T21:56:36.506 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T21:56:36.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T21:56:36.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T21:56:36.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:56:36.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T21:56:36.523 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:56:36.524 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:56:36.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:56:36.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:56:36.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:56:36.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:56:36.837 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:56:36.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:56:36.838 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:56:36.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:56:36.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:56:36.840 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:56:36.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:56:36.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:56:36.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:56:36.843 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:56:36.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:56:36.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:56:36.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T21:56:36.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:56:36.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:36.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:36.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T21:56:36.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:56:36.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:56:36.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:56:36.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:56:36.891 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:56:36.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:56:36.892 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:56:36.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:56:36.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:56:36.894 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:56:36.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:56:36.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:56:36.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:56:36.896 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:56:36.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:56:36.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:56:36.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T21:56:36.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:56:36.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:36.900 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:36.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T21:56:36.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:56:36.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:56:36.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T21:56:36.902 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:56:36.902 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:36.902 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:36.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T21:56:36.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:56:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:56:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:56:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T21:56:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T21:56:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T21:56:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:56:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:56:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:56:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T21:56:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T21:56:37.372 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:56:37.372 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:56:37.372 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:56:37.372 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:56:37.372 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:37.373 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:37.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:56:37.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:56:37.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:56:37.575 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:56:37.575 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:56:37.576 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:56:37.576 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:56:37.576 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:56:37.576 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:56:37.576 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:56:37.577 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:56:37.577 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:56:37.577 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:37.577 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:37.578 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T21:56:37.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:56:37.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T21:56:37.695 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:56:37.695 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:56:37.695 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:56:37.695 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:56:37.695 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:56:37.696 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:56:37.696 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:56:37.696 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:56:37.696 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:56:37.696 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:37.696 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:37.697 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T21:56:37.697 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:56:37.697 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T21:56:37.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T21:56:37.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T21:56:37.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:56:37.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:56:37.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:56:37.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T21:56:37.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:56:38.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:56:38.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:56:38.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:56:38.683 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:56:38.683 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:38.684 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:38.684 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:56:38.684 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:56:38.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:56:38.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T21:56:38.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:56:39.002 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T21:56:39.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:56:40.020 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T21:56:40.020 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:56:40.020 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:56:40.021 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:56:40.021 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:40.021 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:40.021 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T21:56:40.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T21:56:40.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T21:56:40.119 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T21:56:40.131 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T21:56:40.134 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:56:37.413+0000 7fb83fe34e00 0 load: jerasure load: lrc 2024-09-15T21:56:40.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2024-09-15T21:56:40.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2024-09-15T21:56:40.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:439: TEST_ec_recovery_multiple_errors: local objname=myobject 2024-09-15T21:56:40.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: setup_osds 9 2024-09-15T21:56:40.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2024-09-15T21:56:40.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T21:56:40.136 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2024-09-15T21:56:40.137 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2024-09-15T21:56:40.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:56:40.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T21:56:40.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:56:40.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:56:40.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:56:40.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:56:40.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T21:56:40.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:56:40.142 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:56:40.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:56:40.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:56:40.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:56:40.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:56:40.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T21:56:40.144 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:56:40.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=97615de9-aa1c-4634-a371-d61a56fcd3d9 2024-09-15T21:56:40.145 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 97615de9-aa1c-4634-a371-d61a56fcd3d9 2024-09-15T21:56:40.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 97615de9-aa1c-4634-a371-d61a56fcd3d9' 2024-09-15T21:56:40.145 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:56:40.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAYWOdmPeGZCRAAtoTVL94QsJ6tfp112//OSA== 2024-09-15T21:56:40.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAYWOdmPeGZCRAAtoTVL94QsJ6tfp112//OSA=="}' 2024-09-15T21:56:40.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 97615de9-aa1c-4634-a371-d61a56fcd3d9 -i td/test-erasure-eio/0/new.json 2024-09-15T21:56:40.462 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:56:40.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T21:56:40.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAYWOdmPeGZCRAAtoTVL94QsJ6tfp112//OSA== --osd-uuid 97615de9-aa1c-4634-a371-d61a56fcd3d9 2024-09-15T21:56:40.506 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:40.505+0000 7fb6ce3047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:40.531 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:40.529+0000 7fb6ce3047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:40.532 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:40.529+0000 7fb6ce3047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:40.532 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:40.529+0000 7fb6ce3047c0 -1 bdev(0x564bb0f31c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:56:40.532 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:40.529+0000 7fb6ce3047c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T21:56:45.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T21:56:45.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:56:45.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:56:45.859 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T21:56:45.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:56:46.446 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T21:56:46.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:56:46.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:56:46.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:56:46.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:56:46.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:56:46.478 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:46.473+0000 7fa8837087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:46.501 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:46.497+0000 7fa8837087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:46.502 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:46.501+0000 7fa8837087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:46.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:56:46.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:56:46.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:56:46.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:56:46.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:56:46.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:46.874 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:56:46.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:56:46.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:46.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:56:47.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:47.537 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:47.533+0000 7fa8837087c0 -1 Falling back to public interface 2024-09-15T21:56:48.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:48.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:48.336 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:56:48.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:56:48.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:48.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:56:48.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:49.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:49.808 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:56:49.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:49.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:56:49.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:49.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:56:50.166 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:56:50.160+0000 7fa8837087c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:56:50.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:51.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:51.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:51.236 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:56:51.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:56:51.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:51.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:56:51.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:52.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:52.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:52.699 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:56:52.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:56:52.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:52.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:56:53.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:54.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:54.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:54.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:56:54.236 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:56:54.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:54.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:56:54.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:55.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:55.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:55.715 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:56:55.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:56:55.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:55.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:56:56.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:57.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:57.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:57.141 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:56:57.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:56:57.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:57.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:56:57.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:58.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:58.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:58.589 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:56:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:56:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:56:59.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:00.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:00.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:00.155 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:57:00.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:57:00.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:00.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:00.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:01.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:01.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:01.594 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:57:01.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:57:01.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:01.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:02.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:03.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:03.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:03.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:57:03.430 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:57:03.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:03.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:04.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:05.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:05.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:05.295 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:57:05.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:57:05.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:05.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:06.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:07.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:07.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:07.356 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:57:07.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:57:07.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:07.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:08.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:09.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:09.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:09.143 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:57:09.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:57:09.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:09.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:09.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:10.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:10.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:10.637 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:57:10.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:57:10.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:10.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:11.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:12.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:12.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:12.181 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:57:12.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:57:12.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:12.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:12.619 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/985522428,v1:127.0.0.1:6803/985522428] [v2:127.0.0.1:6804/985522428,v1:127.0.0.1:6805/985522428] exists,up 97615de9-aa1c-4634-a371-d61a56fcd3d9 2024-09-15T21:57:12.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:57:12.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:57:12.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:57:12.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:57:12.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T21:57:12.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:57:12.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:57:12.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:57:12.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:57:12.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:57:12.623 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:57:12.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:57:12.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:57:12.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:57:12.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:57:12.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T21:57:12.625 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:57:12.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e7dbd3ad-41c8-47d3-bbe3-81e33192e9d5 2024-09-15T21:57:12.626 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 e7dbd3ad-41c8-47d3-bbe3-81e33192e9d5 2024-09-15T21:57:12.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e7dbd3ad-41c8-47d3-bbe3-81e33192e9d5' 2024-09-15T21:57:12.626 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:57:12.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA4WOdm1GKAJhAAAzgX8rpc4VsoqAiY9HsUGA== 2024-09-15T21:57:12.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA4WOdm1GKAJhAAAzgX8rpc4VsoqAiY9HsUGA=="}' 2024-09-15T21:57:12.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e7dbd3ad-41c8-47d3-bbe3-81e33192e9d5 -i td/test-erasure-eio/1/new.json 2024-09-15T21:57:13.389 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:57:13.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T21:57:13.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4WOdm1GKAJhAAAzgX8rpc4VsoqAiY9HsUGA== --osd-uuid e7dbd3ad-41c8-47d3-bbe3-81e33192e9d5 2024-09-15T21:57:13.435 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:13.432+0000 7fcf281417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:13.460 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:13.456+0000 7fcf281417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:13.462 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:13.460+0000 7fcf281417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:13.462 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:13.460+0000 7fcf281417c0 -1 bdev(0x56089cf7bc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:57:13.462 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:13.460+0000 7fcf281417c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T21:57:19.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T21:57:19.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:57:19.320 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T21:57:19.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:57:19.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:57:19.901 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T21:57:19.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:57:19.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:57:19.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:57:19.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:57:19.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:57:19.933 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:19.928+0000 7f7a4bb257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:19.976 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:19.972+0000 7f7a4bb257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:19.977 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:19.972+0000 7f7a4bb257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:20.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:57:20.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:57:20.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:57:20.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:57:20.345 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:57:20.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:57:20.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:20.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:57:20.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:20.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:20.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:21.493 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:21.488+0000 7f7a4bb257c0 -1 Falling back to public interface 2024-09-15T21:57:21.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:21.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:21.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:57:21.795 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:57:21.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:21.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:22.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:23.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:23.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:23.283 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:57:23.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:57:23.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:23.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:23.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:24.299 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:24.296+0000 7f7a4bb257c0 -1 osd.1 0 log_to_monitors true 2024-09-15T21:57:24.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:24.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:24.801 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:57:24.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:57:24.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:24.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:25.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:26.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:26.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:26.426 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:57:26.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:57:26.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:26.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:26.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:27.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:27.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:27.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:57:27.885 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:57:27.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:27.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:28.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:29.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:29.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:29.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:57:29.375 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:57:29.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:29.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:29.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:30.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:30.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:30.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:57:30.804 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:57:30.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:30.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:31.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:32.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:32.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:32.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:57:32.243 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:57:32.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:32.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:32.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:33.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:33.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:33.670 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:57:33.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:57:33.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:33.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:34.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:35.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:35.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:35.180 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:57:35.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:57:35.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:35.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:35.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:36.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:36.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:36.633 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:57:36.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:57:36.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:36.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:37.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:38.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:38.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:38.523 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:57:38.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:57:38.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:38.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:39.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:40.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:40.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:40.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:57:40.384 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:57:40.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:40.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:41.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:42.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:42.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:42.259 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:57:42.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:57:42.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:42.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:42.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:43.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:43.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:43.938 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:57:43.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:57:43.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:43.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:44.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:45.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:45.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:45.716 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:57:45.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:57:45.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:45.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:46.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:47.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:47.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:47.717 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:57:47.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:57:47.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:47.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:57:48.298 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4021229474,v1:127.0.0.1:6811/4021229474] [v2:127.0.0.1:6812/4021229474,v1:127.0.0.1:6813/4021229474] exists,up e7dbd3ad-41c8-47d3-bbe3-81e33192e9d5 2024-09-15T21:57:48.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:57:48.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:57:48.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:57:48.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:57:48.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:57:48.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:57:48.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:57:48.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:57:48.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:57:48.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:57:48.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:57:48.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:57:48.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:57:48.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T21:57:48.305 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:57:48.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=97f73614-a5e7-4aa2-9bc6-239e4aeeab09 2024-09-15T21:57:48.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 97f73614-a5e7-4aa2-9bc6-239e4aeeab09' 2024-09-15T21:57:48.305 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 97f73614-a5e7-4aa2-9bc6-239e4aeeab09 2024-09-15T21:57:48.305 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:57:48.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBcWOdmCjsjExAAA/3cky1MVIBRrVv5dP+ULQ== 2024-09-15T21:57:48.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBcWOdmCjsjExAAA/3cky1MVIBRrVv5dP+ULQ=="}' 2024-09-15T21:57:48.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 97f73614-a5e7-4aa2-9bc6-239e4aeeab09 -i td/test-erasure-eio/2/new.json 2024-09-15T21:57:48.807 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:57:48.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T21:57:48.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcWOdmCjsjExAAA/3cky1MVIBRrVv5dP+ULQ== --osd-uuid 97f73614-a5e7-4aa2-9bc6-239e4aeeab09 2024-09-15T21:57:48.861 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:48.856+0000 7facdfda07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:48.863 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:48.860+0000 7facdfda07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:48.864 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:48.860+0000 7facdfda07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:48.865 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:48.860+0000 7facdfda07c0 -1 bdev(0x55c0fc163c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:57:48.865 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:48.860+0000 7facdfda07c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T21:57:55.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T21:57:55.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:57:55.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:57:55.158 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T21:57:55.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:57:55.751 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T21:57:55.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:57:55.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:57:55.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:57:55.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:57:55.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:57:55.784 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:55.780+0000 7ffa8bbc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:55.835 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:55.832+0000 7ffa8bbc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:55.836 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:55.832+0000 7ffa8bbc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:56.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:57:56.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:57:56.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:57:56.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:57:56.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:57:56.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:56.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:57:56.276 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:57:56.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:56.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:57:56.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:57.077 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:57:57.072+0000 7ffa8bbc47c0 -1 Falling back to public interface 2024-09-15T21:57:57.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:57.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:57.713 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:57:57.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:57:57.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:57.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:57:58.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:59.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:59.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:59.260 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:57:59.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:57:59.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:59.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:57:59.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:00.093 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:58:00.088+0000 7ffa8bbc47c0 -1 osd.2 0 log_to_monitors true 2024-09-15T21:58:00.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:00.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:00.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:58:00.699 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:58:00.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:00.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:01.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:02.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:02.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:02.218 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:58:02.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:58:02.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:02.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:02.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:03.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:03.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:03.682 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:58:03.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:58:03.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:03.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:04.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:05.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:05.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:05.099 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:58:05.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:58:05.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:05.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:05.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:06.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:06.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:06.519 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:58:06.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:58:06.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:06.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:07.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:08.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:08.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:08.010 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:58:08.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:58:08.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:08.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:08.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:09.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:09.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:09.440 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:58:09.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:58:09.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:09.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:09.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:10.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:10.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:10.994 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:58:10.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:58:10.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:10.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:11.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:12.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:12.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:12.478 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:58:12.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:58:12.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:12.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:13.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:14.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:14.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:14.255 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:58:14.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:58:14.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:14.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:15.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:16.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:16.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:16.301 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:58:16.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:58:16.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:16.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:17.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:18.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:18.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:18.351 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:58:18.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:58:18.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:18.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:19.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:20.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:20.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:20.126 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:58:20.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:58:20.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:20.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:20.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:21.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:21.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:21.699 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:58:21.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:58:21.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:21.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:22.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:23.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:23.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:23.458 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:58:23.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:58:23.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:23.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:58:24.153 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/261859845,v1:127.0.0.1:6819/261859845] [v2:127.0.0.1:6820/261859845,v1:127.0.0.1:6821/261859845] exists,up 97f73614-a5e7-4aa2-9bc6-239e4aeeab09 2024-09-15T21:58:24.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:58:24.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:58:24.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:58:24.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:58:24.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T21:58:24.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:58:24.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:58:24.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T21:58:24.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:58:24.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T21:58:24.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:58:24.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:58:24.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:58:24.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:58:24.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T21:58:24.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T21:58:24.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:58:24.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:58:24.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:58:24.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:58:24.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:58:24.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:58:24.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:58:24.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:58:24.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T21:58:24.160 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:58:24.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c6c7b2c9-8dae-4d96-ba01-6e569b345edf 2024-09-15T21:58:24.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 c6c7b2c9-8dae-4d96-ba01-6e569b345edf' 2024-09-15T21:58:24.161 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 c6c7b2c9-8dae-4d96-ba01-6e569b345edf 2024-09-15T21:58:24.161 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:58:24.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCAWOdmK3q7ChAAYqN5c4rT4ZgzFKAHujWUng== 2024-09-15T21:58:24.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCAWOdmK3q7ChAAYqN5c4rT4ZgzFKAHujWUng=="}' 2024-09-15T21:58:24.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c6c7b2c9-8dae-4d96-ba01-6e569b345edf -i td/test-erasure-eio/3/new.json 2024-09-15T21:58:24.752 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:58:24.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T21:58:24.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCAWOdmK3q7ChAAYqN5c4rT4ZgzFKAHujWUng== --osd-uuid c6c7b2c9-8dae-4d96-ba01-6e569b345edf 2024-09-15T21:58:24.797 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:58:24.795+0000 7f6a4cff07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:58:24.799 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:58:24.799+0000 7f6a4cff07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:58:24.800 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:58:24.799+0000 7f6a4cff07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:58:24.801 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:58:24.799+0000 7f6a4cff07c0 -1 bdev(0x5625c12e7c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T21:58:24.801 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:58:24.799+0000 7f6a4cff07c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T21:58:31.457 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T21:58:31.457 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:58:31.458 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T21:58:31.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T21:58:31.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:58:32.044 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T21:58:32.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T21:58:32.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:58:32.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:58:32.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:58:32.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:58:32.076 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:58:32.075+0000 7fcba910f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:58:32.079 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:58:32.079+0000 7fcba910f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:58:32.080 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:58:32.079+0000 7fcba910f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:58:32.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T21:58:32.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:58:32.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:58:32.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:58:32.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:58:32.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:32.522 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:58:32.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:58:32.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:32.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:33.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:33.333 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:58:33.331+0000 7fcba910f7c0 -1 Falling back to public interface 2024-09-15T21:58:34.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:34.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:34.017 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:58:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:58:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:34.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:35.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:35.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:35.484 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:58:35.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:58:35.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:35.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:36.287 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:58:36.287+0000 7fcba910f7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T21:58:36.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:36.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:36.904 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:58:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:58:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:36.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:37.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:38.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:38.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:38.510 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:58:38.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:58:38.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:38.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:39.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:40.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:40.109 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:58:40.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:40.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:58:40.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:40.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:40.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:41.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:41.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:41.557 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:58:41.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:58:41.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:41.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:42.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:43.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:43.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:43.129 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:58:43.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:58:43.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:43.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:43.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:44.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:44.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:44.670 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:58:44.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:58:44.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:44.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:45.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:46.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:46.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:46.255 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:58:46.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:58:46.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:46.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:46.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:47.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:47.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:47.706 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:58:47.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:58:47.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:47.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:48.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:49.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:49.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:49.201 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:58:49.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:58:49.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:49.202 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:49.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:50.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:50.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:50.655 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:58:50.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:58:50.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:50.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:51.092 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:52.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:52.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:52.094 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:58:52.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:58:52.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:52.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:52.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:53.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:53.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:53.694 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:58:53.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:58:53.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:53.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:54.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:55.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:55.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:55.481 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:58:55.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:58:55.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:55.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:56.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:57.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:57.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:57.679 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:58:57.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:58:57.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:57.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:58:58.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:59.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:59.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:59.353 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:58:59.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:58:59.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:59.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:59:00.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:01.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:01.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:01.080 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:59:01.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:59:01.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:01.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:59:01.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:02.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:02.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:02.747 INFO:tasks.workunit.client.0.smithi104.stdout:19 2024-09-15T21:59:02.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:59:02.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:02.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:59:03.293 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/756186145,v1:127.0.0.1:6827/756186145] [v2:127.0.0.1:6828/756186145,v1:127.0.0.1:6829/756186145] exists,up c6c7b2c9-8dae-4d96-ba01-6e569b345edf 2024-09-15T21:59:03.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:59:03.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:59:03.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:59:03.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:59:03.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T21:59:03.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:59:03.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:59:03.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-15T21:59:03.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:59:03.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T21:59:03.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:59:03.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:59:03.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:59:03.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:59:03.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:59:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:59:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:59:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:59:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:59:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:59:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:59:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:59:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:59:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:59:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T21:59:03.300 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:59:03.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=61b9e3ca-8a8d-429d-a7a0-db068fb7172d 2024-09-15T21:59:03.301 INFO:tasks.workunit.client.0.smithi104.stdout:add osd4 61b9e3ca-8a8d-429d-a7a0-db068fb7172d 2024-09-15T21:59:03.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 61b9e3ca-8a8d-429d-a7a0-db068fb7172d' 2024-09-15T21:59:03.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:59:03.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCnWOdmaXcjExAA47ntKc2XgfaY4DdcmJNR5Q== 2024-09-15T21:59:03.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCnWOdmaXcjExAA47ntKc2XgfaY4DdcmJNR5Q=="}' 2024-09-15T21:59:03.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 61b9e3ca-8a8d-429d-a7a0-db068fb7172d -i td/test-erasure-eio/4/new.json 2024-09-15T21:59:03.995 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:59:04.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T21:59:04.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnWOdmaXcjExAA47ntKc2XgfaY4DdcmJNR5Q== --osd-uuid 61b9e3ca-8a8d-429d-a7a0-db068fb7172d 2024-09-15T21:59:04.048 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:04.047+0000 7f42118a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:04.050 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:04.051+0000 7f42118a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:04.051 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:04.051+0000 7f42118a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:04.052 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:04.051+0000 7f42118a67c0 -1 bdev(0x55afc8253c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-15T21:59:04.052 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:04.051+0000 7f42118a67c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T21:59:10.751 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T21:59:10.751 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:59:10.752 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd4 key to auth repository 2024-09-15T21:59:10.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-15T21:59:10.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:59:11.319 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.4 2024-09-15T21:59:11.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-15T21:59:11.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:59:11.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:59:11.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:59:11.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:59:11.351 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:11.351+0000 7fb09afe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:11.370 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:11.371+0000 7fb09afe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:11.371 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:11.371+0000 7fb09afe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:11.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-15T21:59:11.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:59:11.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-15T21:59:11.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:59:11.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:59:11.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:11.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:59:11.826 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:59:11.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:11.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:12.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:12.641 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:12.639+0000 7fb09afe47c0 -1 Falling back to public interface 2024-09-15T21:59:13.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:13.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:13.321 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:59:13.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:59:13.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:13.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:13.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:14.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:14.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:14.829 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:59:14.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:59:14.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:14.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:15.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:15.401 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:15.399+0000 7fb09afe47c0 -1 osd.4 0 log_to_monitors true 2024-09-15T21:59:16.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:16.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:16.276 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:59:16.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:59:16.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:16.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:17.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:18.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:18.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:18.059 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:59:18.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:59:18.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:18.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:18.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:19.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:19.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:59:19.647 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:59:19.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:19.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:20.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:21.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:21.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:21.214 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T21:59:21.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:59:21.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:21.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:21.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:22.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:22.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:22.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:59:22.647 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T21:59:22.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:22.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:23.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:24.092 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:24.092 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:24.092 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T21:59:24.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:59:24.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:24.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:24.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:25.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:25.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:25.600 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T21:59:25.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:59:25.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:25.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:26.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:27.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:27.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:27.105 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T21:59:27.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:59:27.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:27.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:27.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:28.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:28.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:28.706 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T21:59:28.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:59:28.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:28.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:29.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:30.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:30.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:30.364 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T21:59:30.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:59:30.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:30.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:31.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:32.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:32.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:32.150 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T21:59:32.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:59:32.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:32.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:33.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:34.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:34.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:34.040 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T21:59:34.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:59:34.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:34.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:35.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:36.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:36.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:36.296 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T21:59:36.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:59:36.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:36.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:36.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:37.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:37.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:37.919 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T21:59:37.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:59:37.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:37.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:38.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:39.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:39.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:39.708 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T21:59:39.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:59:39.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:39.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:40.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:41.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:41.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:41.608 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T21:59:41.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:59:41.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:41.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:59:42.308 INFO:tasks.workunit.client.0.smithi104.stdout:osd.4 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1735632671,v1:127.0.0.1:6835/1735632671] [v2:127.0.0.1:6836/1735632671,v1:127.0.0.1:6837/1735632671] exists,up 61b9e3ca-8a8d-429d-a7a0-db068fb7172d 2024-09-15T21:59:42.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:59:42.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:59:42.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:59:42.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T21:59:42.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-15T21:59:42.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T21:59:42.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:59:42.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-15T21:59:42.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:59:42.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-15T21:59:42.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T21:59:42.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:59:42.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:59:42.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:59:42.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-15T21:59:42.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-15T21:59:42.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:59:42.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:59:42.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T21:59:42.314 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:59:42.315 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:59:42.315 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:59:42.315 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:59:42.315 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:59:42.315 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T21:59:42.315 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:59:42.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T21:59:42.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:59:42.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:59:42.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:59:42.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T21:59:42.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T21:59:42.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:59:42.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:59:42.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:59:42.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:59:42.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:59:42.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:59:42.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-15T21:59:42.317 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:59:42.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ba4a30fc-d3ad-4ad1-b406-3f571e3d469b 2024-09-15T21:59:42.318 INFO:tasks.workunit.client.0.smithi104.stdout:add osd5 ba4a30fc-d3ad-4ad1-b406-3f571e3d469b 2024-09-15T21:59:42.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 ba4a30fc-d3ad-4ad1-b406-3f571e3d469b' 2024-09-15T21:59:42.318 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:59:42.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDOWOdmpyAxFBAACsit8/wxUU+QcSMM+be2zQ== 2024-09-15T21:59:42.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDOWOdmpyAxFBAACsit8/wxUU+QcSMM+be2zQ=="}' 2024-09-15T21:59:42.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ba4a30fc-d3ad-4ad1-b406-3f571e3d469b -i td/test-erasure-eio/5/new.json 2024-09-15T21:59:43.078 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:59:43.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-15T21:59:43.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOWOdmpyAxFBAACsit8/wxUU+QcSMM+be2zQ== --osd-uuid ba4a30fc-d3ad-4ad1-b406-3f571e3d469b 2024-09-15T21:59:43.134 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:43.131+0000 7feb3e2e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:43.185 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:43.183+0000 7feb3e2e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:43.185 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:43.183+0000 7feb3e2e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:43.186 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:43.187+0000 7feb3e2e07c0 -1 bdev(0x5582ebcf9c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-15T21:59:43.186 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:43.187+0000 7feb3e2e07c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-15T21:59:49.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-15T21:59:49.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:59:49.708 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd5 key to auth repository 2024-09-15T21:59:49.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-15T21:59:49.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:59:50.282 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.5 2024-09-15T21:59:50.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-15T21:59:50.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:59:50.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:59:50.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:59:50.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:59:50.314 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:50.310+0000 7f1d0bf9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:50.335 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:50.334+0000 7f1d0bf9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:50.336 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:50.334+0000 7f1d0bf9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:50.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-15T21:59:50.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:59:50.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:59:50.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:59:50.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:59:50.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:50.722 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T21:59:50.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:59:50.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:50.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:59:51.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:51.625 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:51.622+0000 7f1d0bf9d7c0 -1 Falling back to public interface 2024-09-15T21:59:52.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:52.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:52.211 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T21:59:52.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:59:52.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:52.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:59:52.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:53.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:53.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:53.678 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T21:59:53.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:59:53.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:53.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:59:54.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:54.581 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T21:59:54.578+0000 7f1d0bf9d7c0 -1 osd.5 0 log_to_monitors true 2024-09-15T21:59:55.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:55.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:55.223 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T21:59:55.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:59:55.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:55.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:59:55.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:56.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:56.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:56.981 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T21:59:56.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:59:56.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:56.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:59:57.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:58.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:58.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:58.777 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T21:59:58.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:59:58.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:58.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:59:59.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:00.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:00.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:00.215 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:00:00.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:00:00.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:00.216 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:00.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:01.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:01.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:01.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:00:01.764 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:00:01.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:01.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:02.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:03.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:03.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:03.261 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:00:03.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:00:03.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:03.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:03.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:04.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:04.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:04.724 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:00:04.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:00:04.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:04.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:05.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:06.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:06.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:06.167 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:00:06.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:00:06.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:06.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:06.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:07.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:07.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:07.608 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:00:07.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:00:07.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:07.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:08.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:09.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:09.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:09.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:00:09.127 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:00:09.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:09.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:09.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:10.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:10.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:10.580 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:00:10.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:00:10.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:10.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:11.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:12.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:12.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:12.061 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:00:12.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:00:12.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:12.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:12.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:13.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:13.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:13.641 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:00:13.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:00:13.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:13.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:14.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:15.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:15.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:15.715 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:00:15.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:00:15.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:15.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:16.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:17.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:17.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:17.175 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:00:17.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:00:17.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:17.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:18.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:19.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:19.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:19.083 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:00:19.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:00:19.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:19.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:00:19.646 INFO:tasks.workunit.client.0.smithi104.stdout:osd.5 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2805941663,v1:127.0.0.1:6843/2805941663] [v2:127.0.0.1:6844/2805941663,v1:127.0.0.1:6845/2805941663] exists,up ba4a30fc-d3ad-4ad1-b406-3f571e3d469b 2024-09-15T22:00:19.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:00:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:00:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:00:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:00:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-15T22:00:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:00:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:00:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-15T22:00:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:00:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-15T22:00:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:00:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:00:19.651 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:00:19.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:00:19.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:00:19.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-15T22:00:19.653 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:00:19.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f6eb460e-af0a-4006-84fb-0821220f8f53 2024-09-15T22:00:19.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 f6eb460e-af0a-4006-84fb-0821220f8f53' 2024-09-15T22:00:19.654 INFO:tasks.workunit.client.0.smithi104.stdout:add osd6 f6eb460e-af0a-4006-84fb-0821220f8f53 2024-09-15T22:00:19.654 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:00:19.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDzWOdmDZ4wKBAArL1B7GM5t8huXwsZDS6cVg== 2024-09-15T22:00:19.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDzWOdmDZ4wKBAArL1B7GM5t8huXwsZDS6cVg=="}' 2024-09-15T22:00:19.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f6eb460e-af0a-4006-84fb-0821220f8f53 -i td/test-erasure-eio/6/new.json 2024-09-15T22:00:20.331 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:00:20.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-15T22:00:20.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzWOdmDZ4wKBAArL1B7GM5t8huXwsZDS6cVg== --osd-uuid f6eb460e-af0a-4006-84fb-0821220f8f53 2024-09-15T22:00:20.381 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:20.378+0000 7f801933c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:20.470 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:20.470+0000 7f801933c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:20.472 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:20.470+0000 7f801933c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:20.472 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:20.470+0000 7f801933c7c0 -1 bdev(0x561a81d49c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-15T22:00:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:20.470+0000 7f801933c7c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-15T22:00:27.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-15T22:00:27.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:00:27.381 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd6 key to auth repository 2024-09-15T22:00:27.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-15T22:00:27.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:00:27.973 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.6 2024-09-15T22:00:27.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-15T22:00:27.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:00:27.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:00:27.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:00:27.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:00:28.006 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:28.006+0000 7f1484dcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:28.008 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:28.006+0000 7f1484dcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:28.010 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:28.006+0000 7f1484dcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:28.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-15T22:00:28.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:00:28.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-15T22:00:28.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:00:28.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:00:28.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:28.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:00:28.485 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:00:28.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:28.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:28.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:29.477 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:29.474+0000 7f1484dcb7c0 -1 Falling back to public interface 2024-09-15T22:00:29.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:29.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:29.969 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:00:29.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:00:29.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:29.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:30.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:31.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:31.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:31.446 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:00:31.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:00:31.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:31.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:31.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:32.274 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:32.270+0000 7f1484dcb7c0 -1 osd.6 0 log_to_monitors true 2024-09-15T22:00:32.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:32.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:32.886 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:00:32.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:00:32.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:32.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:33.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:34.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:34.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:34.654 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:00:34.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:00:34.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:34.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:35.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:36.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:36.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:36.212 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:00:36.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:00:36.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:36.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:36.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:37.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:37.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:37.690 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:00:37.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:00:37.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:37.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:38.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:39.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:39.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:39.265 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:00:39.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:00:39.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:39.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:39.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:40.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:40.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:40.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:00:40.709 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:00:40.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:40.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:41.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:42.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:42.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:42.168 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:00:42.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:00:42.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:42.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:42.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:43.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:43.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:43.625 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:00:43.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:00:43.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:43.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:44.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:45.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:45.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:45.124 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:00:45.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:00:45.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:45.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:45.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:46.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:46.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:46.832 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:00:46.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:00:46.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:46.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:47.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:48.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:48.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:48.774 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:00:48.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:00:48.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:48.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:49.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:50.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:50.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:50.542 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:00:50.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:00:50.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:50.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:51.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:52.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:52.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:52.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:00:52.520 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:00:52.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:52.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:53.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:54.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:54.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:54.029 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:00:54.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:00:54.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:54.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:54.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:55.467 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:00:55.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:55.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:55.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:00:55.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:55.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:00:56.005 INFO:tasks.workunit.client.0.smithi104.stdout:osd.6 up in weight 1 up_from 59 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1287108356,v1:127.0.0.1:6851/1287108356] [v2:127.0.0.1:6852/1287108356,v1:127.0.0.1:6853/1287108356] exists,up f6eb460e-af0a-4006-84fb-0821220f8f53 2024-09-15T22:00:56.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:00:56.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:00:56.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:00:56.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:00:56.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2024-09-15T22:00:56.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:00:56.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:00:56.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-15T22:00:56.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:00:56.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/7 2024-09-15T22:00:56.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:00:56.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:00:56.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:00:56.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:00:56.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2024-09-15T22:00:56.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2024-09-15T22:00:56.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:00:56.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:00:56.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:00:56.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:00:56.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:00:56.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:00:56.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:00:56.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:00:56.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:00:56.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:00:56.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:00:56.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/7 2024-09-15T22:00:56.012 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:00:56.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e2d49ce1-6609-47fd-8e0e-4e0ff8a3e67f 2024-09-15T22:00:56.013 INFO:tasks.workunit.client.0.smithi104.stdout:add osd7 e2d49ce1-6609-47fd-8e0e-4e0ff8a3e67f 2024-09-15T22:00:56.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 e2d49ce1-6609-47fd-8e0e-4e0ff8a3e67f' 2024-09-15T22:00:56.014 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:00:56.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAYWedmG5AMAhAAKqRYI9nqebXHtvNCE/q6rQ== 2024-09-15T22:00:56.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAYWedmG5AMAhAAKqRYI9nqebXHtvNCE/q6rQ=="}' 2024-09-15T22:00:56.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e2d49ce1-6609-47fd-8e0e-4e0ff8a3e67f -i td/test-erasure-eio/7/new.json 2024-09-15T22:00:56.589 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:00:56.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/7/new.json 2024-09-15T22:00:56.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAYWedmG5AMAhAAKqRYI9nqebXHtvNCE/q6rQ== --osd-uuid e2d49ce1-6609-47fd-8e0e-4e0ff8a3e67f 2024-09-15T22:00:56.641 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:56.638+0000 7f2640ae07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:56.745 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:56.742+0000 7f2640ae07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:56.747 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:56.746+0000 7f2640ae07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:56.748 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:56.746+0000 7f2640ae07c0 -1 bdev(0x55b847eadc00 td/test-erasure-eio/7/block) open stat got: (1) Operation not permitted 2024-09-15T22:00:56.748 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:00:56.746+0000 7f2640ae07c0 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2024-09-15T22:01:03.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2024-09-15T22:01:03.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:01:03.129 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd7 key to auth repository 2024-09-15T22:01:03.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-15T22:01:03.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:01:03.697 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.7 2024-09-15T22:01:03.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-15T22:01:03.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:01:03.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:01:03.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:01:03.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:01:03.729 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:03.726+0000 7f0c9df197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:03.734 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:03.734+0000 7f0c9df197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:03.735 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:03.734+0000 7f0c9df197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:04.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-15T22:01:04.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:01:04.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-15T22:01:04.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:01:04.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:01:04.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:04.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:01:04.182 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:01:04.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:04.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:04.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:05.297 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:05.293+0000 7f0c9df197c0 -1 Falling back to public interface 2024-09-15T22:01:05.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:05.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:05.660 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:01:05.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:01:05.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:05.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:06.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:07.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:07.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:07.103 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:01:07.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:01:07.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:07.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:07.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:08.236 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:08.233+0000 7f0c9df197c0 -1 osd.7 0 log_to_monitors true 2024-09-15T22:01:08.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:08.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:08.576 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:01:08.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:01:08.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:08.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:09.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:10.019 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:01:10.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:10.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:10.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:01:10.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:10.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:10.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:11.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:11.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:11.683 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:01:11.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:01:11.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:11.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:12.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:13.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:13.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:13.196 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:01:13.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:01:13.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:13.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:13.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:14.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:14.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:14.683 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:01:14.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:01:14.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:14.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:15.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:16.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:16.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:16.128 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:01:16.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:01:16.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:16.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:16.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:17.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:17.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:17.550 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:01:17.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:01:17.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:17.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:18.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:19.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:19.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:19.066 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:01:19.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:01:19.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:19.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:19.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:20.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:20.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:20.503 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:01:20.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:01:20.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:20.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:21.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:22.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:22.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:22.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:01:22.072 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:01:22.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:22.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:22.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:23.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:23.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:23.497 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:01:23.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:01:23.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:23.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:24.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:25.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:25.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:25.039 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:01:25.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:01:25.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:25.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:25.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:26.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:26.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:26.843 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:01:26.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:01:26.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:26.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:27.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:28.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:28.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:28.667 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:01:28.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:01:28.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:28.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:29.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:30.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:30.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:30.922 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:01:30.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:01:30.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:30.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:31.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:32.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:32.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:32.806 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:01:32.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:01:32.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:32.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T22:01:33.492 INFO:tasks.workunit.client.0.smithi104.stdout:osd.7 up in weight 1 up_from 67 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/732677981,v1:127.0.0.1:6859/732677981] [v2:127.0.0.1:6860/732677981,v1:127.0.0.1:6861/732677981] exists,up e2d49ce1-6609-47fd-8e0e-4e0ff8a3e67f 2024-09-15T22:01:33.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:01:33.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:01:33.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:01:33.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/8 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2024-09-15T22:01:33.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:01:33.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:01:33.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:01:33.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:01:33.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:01:33.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:01:33.496 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:01:33.496 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:01:33.496 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:01:33.496 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:01:33.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:01:33.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:01:33.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:01:33.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:01:33.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:01:33.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:01:33.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:01:33.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:01:33.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:01:33.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:01:33.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:01:33.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:01:33.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/8 2024-09-15T22:01:33.497 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:01:33.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=08fd6b3b-5b24-4141-8b9a-46ba20d43a64 2024-09-15T22:01:33.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 08fd6b3b-5b24-4141-8b9a-46ba20d43a64' 2024-09-15T22:01:33.498 INFO:tasks.workunit.client.0.smithi104.stdout:add osd8 08fd6b3b-5b24-4141-8b9a-46ba20d43a64 2024-09-15T22:01:33.498 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:01:33.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA9Wedmby0UHxAAfqPeQQTGAmytI+ZhZyrGYw== 2024-09-15T22:01:33.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA9Wedmby0UHxAAfqPeQQTGAmytI+ZhZyrGYw=="}' 2024-09-15T22:01:33.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 08fd6b3b-5b24-4141-8b9a-46ba20d43a64 -i td/test-erasure-eio/8/new.json 2024-09-15T22:01:34.515 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:01:34.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/8/new.json 2024-09-15T22:01:34.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9Wedmby0UHxAAfqPeQQTGAmytI+ZhZyrGYw== --osd-uuid 08fd6b3b-5b24-4141-8b9a-46ba20d43a64 2024-09-15T22:01:34.567 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:34.565+0000 7f9983b217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:34.570 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:34.565+0000 7f9983b217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:34.571 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:34.569+0000 7f9983b217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:34.571 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:34.569+0000 7f9983b217c0 -1 bdev(0x55cda0959c00 td/test-erasure-eio/8/block) open stat got: (1) Operation not permitted 2024-09-15T22:01:34.571 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:34.569+0000 7f9983b217c0 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2024-09-15T22:01:41.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2024-09-15T22:01:41.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:01:41.510 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd8 key to auth repository 2024-09-15T22:01:41.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-15T22:01:41.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:01:42.113 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.8 2024-09-15T22:01:42.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-15T22:01:42.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:01:42.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:01:42.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:01:42.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:01:42.145 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:42.141+0000 7fb2e53e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:42.159 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:42.157+0000 7fb2e53e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:42.161 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:42.157+0000 7fb2e53e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:42.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-15T22:01:42.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:01:42.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-15T22:01:42.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:01:42.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:01:42.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:42.580 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:01:42.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:01:42.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:01:42.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:43.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:43.237 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:43.233+0000 7fb2e53e47c0 -1 Falling back to public interface 2024-09-15T22:01:44.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:44.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:44.012 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:01:44.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:01:44.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:44.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:01:44.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:45.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:45.429 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:01:45.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:45.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:01:45.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:45.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:01:45.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:46.051 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:01:46.049+0000 7fb2e53e47c0 -1 osd.8 0 log_to_monitors true 2024-09-15T22:01:46.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:46.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:46.873 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:01:46.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:01:46.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:46.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:01:47.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:48.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:48.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:48.882 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:01:48.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:01:48.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:48.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:01:49.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:50.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:50.539 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:50.539 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:01:50.539 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:01:50.539 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:50.539 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:01:51.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:52.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:52.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:52.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:01:52.111 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:01:52.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:52.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:01:52.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:53.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:53.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:53.713 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:01:53.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:01:53.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:53.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:01:54.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:55.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:55.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:55.159 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:01:55.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:01:55.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:55.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:01:55.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:56.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:56.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:56.587 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:01:56.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:01:56.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:56.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:01:57.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:58.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:58.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:58.114 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:01:58.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:01:58.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:58.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:01:58.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:59.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:59.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:59.633 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:01:59.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:01:59.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:59.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:02:00.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:01.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:01.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:01.161 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:02:01.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:02:01.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:01.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:02:01.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:02.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:02.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:02.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:02:02.839 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:02:02.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:02.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:02:03.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:04.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:04.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:04.575 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:02:04.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:02:04.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:04.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:02:05.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:06.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:06.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:06.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:02:06.272 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:02:06.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:06.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:02:07.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:08.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:08.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:08.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:02:08.051 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:02:08.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:08.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:02:08.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:09.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:09.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:09.791 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:02:09.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:02:09.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:09.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:02:10.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:11.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:11.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:11.604 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:02:11.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:02:11.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:11.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T22:02:12.678 INFO:tasks.workunit.client.0.smithi104.stdout:osd.8 up in weight 1 up_from 78 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1016659000,v1:127.0.0.1:6867/1016659000] [v2:127.0.0.1:6868/1016659000,v1:127.0.0.1:6869/1016659000] exists,up 08fd6b3b-5b24-4141-8b9a-46ba20d43a64 2024-09-15T22:02:12.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:02:12.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:02:12.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:02:12.679 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:02:12.679 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:02:12.679 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:02:12.680 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:12.680 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:12.680 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:12.680 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:02:12.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:02:12.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:02:12.795 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:02:12.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:02:12.817 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T21:56:47.545+0000 7fa8837087c0 0 load: jerasure load: lrc 2024-09-15T22:02:12.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2024-09-15T22:02:12.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:444: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2024-09-15T22:02:12.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:02:12.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:02:12.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2024-09-15T22:02:12.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:02:12.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2024-09-15T22:02:12.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:02:12.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2024-09-15T22:02:13.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:02:13.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:02:14.839 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:02:14.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:02:15.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:02:15.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:02:15.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:02:15.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:02:15.860 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:02:15.861 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:02:15.861 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:02:15.862 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:02:15.862 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:02:15.862 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:02:15.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:02:15.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:02:15.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:02:15.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:02:15.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:02:15.987 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:02:16.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:02:16.426 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:02:16.427 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:02:16.427 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:02:16.427 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:02:16.427 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:02:16.427 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T22:02:16.427 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T22:02:16.427 INFO:tasks.workunit.client.0.smithi104.stderr:8' 2024-09-15T22:02:16.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:02:16.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:16.427 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:02:16.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803843 2024-09-15T22:02:16.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803843 2024-09-15T22:02:16.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843' 2024-09-15T22:02:16.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:16.682 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:02:16.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607611 2024-09-15T22:02:16.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607611 2024-09-15T22:02:16.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-51539607611' 2024-09-15T22:02:16.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:16.831 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:02:16.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411381 2024-09-15T22:02:16.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411381 2024-09-15T22:02:16.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-51539607611 2-77309411381' 2024-09-15T22:02:16.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:16.977 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:02:17.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084336 2024-09-15T22:02:17.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084336 2024-09-15T22:02:17.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-51539607611 2-77309411381 3-120259084336' 2024-09-15T22:02:17.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:17.130 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T22:02:17.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724582 2024-09-15T22:02:17.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724582 2024-09-15T22:02:17.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-51539607611 2-77309411381 3-120259084336 4-167503724582' 2024-09-15T22:02:17.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:17.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T22:02:17.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364834 2024-09-15T22:02:17.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364834 2024-09-15T22:02:17.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-51539607611 2-77309411381 3-120259084336 4-167503724582 5-214748364834' 2024-09-15T22:02:17.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:17.439 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T22:02:17.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=253403070491 2024-09-15T22:02:17.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 253403070491 2024-09-15T22:02:17.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-51539607611 2-77309411381 3-120259084336 4-167503724582 5-214748364834 6-253403070491' 2024-09-15T22:02:17.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:17.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T22:02:17.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808849 2024-09-15T22:02:17.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808849 2024-09-15T22:02:17.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-51539607611 2-77309411381 3-120259084336 4-167503724582 5-214748364834 6-253403070491 7-287762808849' 2024-09-15T22:02:17.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:17.757 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449096 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449096 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-51539607611 2-77309411381 3-120259084336 4-167503724582 5-214748364834 6-253403070491 7-287762808849 8-335007449096' 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803843 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803843 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803843 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803843 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803843' 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:02:18.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803843 -lt 25769803843 2024-09-15T22:02:18.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:18.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607611 2024-09-15T22:02:18.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:18.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:02:18.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607611 2024-09-15T22:02:18.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:18.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607611 2024-09-15T22:02:18.375 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 51539607611 2024-09-15T22:02:18.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607611' 2024-09-15T22:02:18.375 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:02:18.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607611 -lt 51539607611 2024-09-15T22:02:18.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:18.863 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411381 2024-09-15T22:02:18.863 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:18.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:02:18.866 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411381 2024-09-15T22:02:18.866 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:18.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411381 2024-09-15T22:02:18.867 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 77309411381 2024-09-15T22:02:18.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411381' 2024-09-15T22:02:18.868 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:02:19.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411381 -lt 77309411381 2024-09-15T22:02:19.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:19.357 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084336 2024-09-15T22:02:19.357 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:19.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:02:19.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084336 2024-09-15T22:02:19.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:19.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084336 2024-09-15T22:02:19.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084336' 2024-09-15T22:02:19.361 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 120259084336 2024-09-15T22:02:19.361 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:02:19.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084337 -lt 120259084336 2024-09-15T22:02:19.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:19.838 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-167503724582 2024-09-15T22:02:19.839 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:19.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T22:02:19.841 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-167503724582 2024-09-15T22:02:19.841 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:19.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724582 2024-09-15T22:02:19.843 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 167503724582 2024-09-15T22:02:19.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 167503724582' 2024-09-15T22:02:19.844 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T22:02:20.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724582 -lt 167503724582 2024-09-15T22:02:20.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:20.262 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-214748364834 2024-09-15T22:02:20.263 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:20.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T22:02:20.265 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-214748364834 2024-09-15T22:02:20.265 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:20.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364834 2024-09-15T22:02:20.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 214748364834' 2024-09-15T22:02:20.266 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 214748364834 2024-09-15T22:02:20.266 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T22:02:20.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364834 -lt 214748364834 2024-09-15T22:02:20.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:20.796 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-253403070491 2024-09-15T22:02:20.796 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:20.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T22:02:20.798 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-253403070491 2024-09-15T22:02:20.798 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:20.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=253403070491 2024-09-15T22:02:20.799 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 253403070491 2024-09-15T22:02:20.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 253403070491' 2024-09-15T22:02:20.800 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T22:02:21.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 253403070491 -lt 253403070491 2024-09-15T22:02:21.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:21.251 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:21.251 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-287762808849 2024-09-15T22:02:21.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T22:02:21.253 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-287762808849 2024-09-15T22:02:21.253 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:21.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808849 2024-09-15T22:02:21.255 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.7 seq 287762808849 2024-09-15T22:02:21.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 287762808849' 2024-09-15T22:02:21.255 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T22:02:21.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808849 -lt 287762808849 2024-09-15T22:02:21.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:21.701 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-335007449096 2024-09-15T22:02:21.702 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:21.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T22:02:21.702 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-335007449096 2024-09-15T22:02:21.702 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:21.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449096 2024-09-15T22:02:21.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 335007449096' 2024-09-15T22:02:21.702 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.8 seq 335007449096 2024-09-15T22:02:21.702 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T22:02:22.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449096 -lt 335007449096 2024-09-15T22:02:22.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:02:22.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:02:22.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:02:22.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:02:22.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:02:22.687 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:02:22.687 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:02:22.687 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:02:22.688 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:02:22.688 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:02:22.688 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:02:23.131 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:02:23.132 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:02:23.132 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:02:23.132 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:02:23.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:02:23.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:02:23.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:02:23.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:446: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-15T22:02:23.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:02:23.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:02:23.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-15T22:02:23.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:02:23.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:02:23.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:02:23.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:02:23.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:02:23.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:02:23.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:02:23.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:02:23.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-15T22:02:23.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:447: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T22:02:23.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:02:23.967 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-15T22:02:23.967 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:02:23.967 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-15T22:02:23.967 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T22:02:23.967 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:02:24.425 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:02:24.425 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:02:24.425 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:02:24.425 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:02:24.425 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T22:02:24.425 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:02:24.425 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:02:24.426 INFO:tasks.workunit.client.0.smithi104.stderr:7' 2024-09-15T22:02:24.426 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-15T22:02:24.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-15T22:02:24.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:02:24.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-15T22:02:24.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:02:24.426 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T22:02:24.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:02:24.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T22:02:24.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:02:24.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-15T22:02:24.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:02:24.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:02:24.428 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:02:24.428 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-15T22:02:24.428 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:02:24.428 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:02:24.428 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:24.429 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:24.429 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:24.429 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:02:24.429 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:02:24.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:02:24.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:02:24.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:02:24.547 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-15T22:02:24.547 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:02:24.547 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:02:24.548 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:24.548 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:24.548 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:24.548 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:02:24.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:02:24.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-15T22:02:24.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:450: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=3 2024-09-15T22:02:24.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:02:24.669 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-15T22:02:24.669 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:02:24.669 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-15T22:02:24.669 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T22:02:24.669 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:02:25.149 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr:7' 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-15T22:02:25.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:02:25.151 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/0/type 2024-09-15T22:02:25.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:02:25.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-15T22:02:25.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:02:25.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-15T22:02:25.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:02:25.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:02:25.152 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:02:25.152 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-15T22:02:25.152 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:02:25.153 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:02:25.153 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:25.153 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:25.153 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:25.153 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:02:25.154 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:02:25.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:02:25.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:02:25.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:02:25.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-15T22:02:25.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:02:25.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:02:25.271 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:25.271 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:25.271 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:25.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:02:25.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:02:25.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2024-09-15T22:02:25.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=4 2024-09-15T22:02:25.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:02:25.381 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-15T22:02:25.381 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:02:25.381 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-15T22:02:25.381 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T22:02:25.381 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:02:25.832 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:02:25.832 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:02:25.832 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:02:25.833 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:02:25.833 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T22:02:25.833 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:02:25.833 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:02:25.833 INFO:tasks.workunit.client.0.smithi104.stderr:7' 2024-09-15T22:02:25.833 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-15T22:02:25.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-15T22:02:25.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:02:25.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=6 2024-09-15T22:02:25.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:02:25.835 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/6/type 2024-09-15T22:02:25.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:02:25.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2024-09-15T22:02:25.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:02:25.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=6 2024-09-15T22:02:25.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:02:25.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:02:25.836 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:02:25.836 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.6 2024-09-15T22:02:25.836 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.6 2024-09-15T22:02:25.836 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.6 ']' 2024-09-15T22:02:25.836 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:25.836 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:25.836 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:25.836 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.6.asok 2024-09-15T22:02:25.836 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:02:25.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:02:25.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:02:25.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:02:25.959 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.6 2024-09-15T22:02:25.959 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.6 2024-09-15T22:02:25.959 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.6 ']' 2024-09-15T22:02:25.960 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:25.960 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:25.960 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:25.960 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.6.asok 2024-09-15T22:02:25.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:02:25.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2024-09-15T22:02:26.071 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2024-09-15T22:02:26.071 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:02:26.072 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-15T22:02:26.072 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T22:02:26.072 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:02:26.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:02:26.555 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:02:26.555 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:02:26.555 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:02:26.555 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T22:02:26.555 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:02:26.555 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:02:26.556 INFO:tasks.workunit.client.0.smithi104.stderr:7' 2024-09-15T22:02:26.556 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-15T22:02:26.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-15T22:02:26.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: local -a initial_osds 2024-09-15T22:02:26.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_errors: local last_osd=7 2024-09-15T22:02:26.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2024-09-15T22:02:26.556 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:02:26.556 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:02:26.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:02:26.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:02:26.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:02:26.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:02:26.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_errors: ceph osd down 7 2024-09-15T22:02:27.572 INFO:tasks.workunit.client.0.smithi104.stderr:osd.7 is already down. 2024-09-15T22:02:27.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:458: TEST_ec_recovery_multiple_errors: ceph osd out 7 2024-09-15T22:02:28.443 INFO:tasks.workunit.client.0.smithi104.stderr:osd.7 is already out. 2024-09-15T22:02:28.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:461: TEST_ec_recovery_multiple_errors: wait_for_clean 2024-09-15T22:02:28.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:02:28.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:02:28.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:02:28.461 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:02:28.462 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:02:28.462 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:02:28.462 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:02:28.462 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:02:28.462 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:02:28.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:02:28.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:02:28.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:02:28.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:02:28.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:02:28.561 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:02:29.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:02:29.087 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:02:29.087 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:02:29.087 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:02:29.087 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:02:29.088 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:02:29.088 INFO:tasks.workunit.client.0.smithi104.stderr:6 2024-09-15T22:02:29.088 INFO:tasks.workunit.client.0.smithi104.stderr:7 2024-09-15T22:02:29.088 INFO:tasks.workunit.client.0.smithi104.stderr:8' 2024-09-15T22:02:29.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:02:29.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:29.088 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:02:29.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803848 2024-09-15T22:02:29.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803848 2024-09-15T22:02:29.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848' 2024-09-15T22:02:29.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:29.253 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:02:29.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607616 2024-09-15T22:02:29.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607616 2024-09-15T22:02:29.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848 1-51539607616' 2024-09-15T22:02:29.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:29.551 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:02:29.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411386 2024-09-15T22:02:29.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411386 2024-09-15T22:02:29.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848 1-51539607616 2-77309411386' 2024-09-15T22:02:29.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:29.704 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:02:30.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084341 2024-09-15T22:02:30.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084341 2024-09-15T22:02:30.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848 1-51539607616 2-77309411386 3-120259084341' 2024-09-15T22:02:30.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:30.259 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T22:02:30.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724586 2024-09-15T22:02:30.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724586 2024-09-15T22:02:30.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848 1-51539607616 2-77309411386 3-120259084341 4-167503724586' 2024-09-15T22:02:30.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:30.427 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T22:02:30.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364839 2024-09-15T22:02:30.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364839 2024-09-15T22:02:30.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848 1-51539607616 2-77309411386 3-120259084341 4-167503724586 5-214748364839' 2024-09-15T22:02:30.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:30.594 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T22:02:30.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=253403070496 2024-09-15T22:02:30.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 253403070496 2024-09-15T22:02:30.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848 1-51539607616 2-77309411386 3-120259084341 4-167503724586 5-214748364839 6-253403070496' 2024-09-15T22:02:30.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:30.753 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T22:02:30.866 INFO:tasks.workunit.client.0.smithi104.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-15T22:02:30.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T22:02:30.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T22:02:30.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T22:02:30.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:30.869 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T22:02:31.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449100 2024-09-15T22:02:31.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449100 2024-09-15T22:02:31.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848 1-51539607616 2-77309411386 3-120259084341 4-167503724586 5-214748364839 6-253403070496 8-335007449100' 2024-09-15T22:02:31.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:31.023 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803848 2024-09-15T22:02:31.023 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:31.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:02:31.025 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803848 2024-09-15T22:02:31.025 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:31.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803848 2024-09-15T22:02:31.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803848' 2024-09-15T22:02:31.026 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803848 2024-09-15T22:02:31.026 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:02:31.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803848 -lt 25769803848 2024-09-15T22:02:31.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:31.445 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:31.445 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607616 2024-09-15T22:02:31.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:02:31.447 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607616 2024-09-15T22:02:31.447 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:31.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607616 2024-09-15T22:02:31.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607616' 2024-09-15T22:02:31.448 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 51539607616 2024-09-15T22:02:31.449 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:02:31.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607616 -lt 51539607616 2024-09-15T22:02:31.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:31.900 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:31.900 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411386 2024-09-15T22:02:31.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:02:31.901 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411386 2024-09-15T22:02:31.901 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:31.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411386 2024-09-15T22:02:31.903 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 77309411386 2024-09-15T22:02:31.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411386' 2024-09-15T22:02:31.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:02:32.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411386 -lt 77309411386 2024-09-15T22:02:32.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:32.336 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084341 2024-09-15T22:02:32.336 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:32.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:02:32.338 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084341 2024-09-15T22:02:32.338 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:32.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084341 2024-09-15T22:02:32.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084341' 2024-09-15T22:02:32.340 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 120259084341 2024-09-15T22:02:32.340 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:02:32.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084341 -lt 120259084341 2024-09-15T22:02:32.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:32.824 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-167503724586 2024-09-15T22:02:32.824 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:32.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T22:02:32.825 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-167503724586 2024-09-15T22:02:32.826 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:32.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724586 2024-09-15T22:02:32.827 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 167503724586 2024-09-15T22:02:32.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 167503724586' 2024-09-15T22:02:32.828 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T22:02:33.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724586 -lt 167503724586 2024-09-15T22:02:33.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:33.265 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-214748364839 2024-09-15T22:02:33.266 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:33.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T22:02:33.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-214748364839 2024-09-15T22:02:33.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:33.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364839 2024-09-15T22:02:33.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 214748364839' 2024-09-15T22:02:33.270 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 214748364839 2024-09-15T22:02:33.270 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T22:02:33.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364839 -lt 214748364839 2024-09-15T22:02:33.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:33.707 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-253403070496 2024-09-15T22:02:33.708 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:33.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T22:02:33.710 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-253403070496 2024-09-15T22:02:33.710 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:33.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=253403070496 2024-09-15T22:02:33.711 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 253403070496 2024-09-15T22:02:33.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 253403070496' 2024-09-15T22:02:33.711 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T22:02:34.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 253403070496 -lt 253403070496 2024-09-15T22:02:34.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:34.144 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-335007449100 2024-09-15T22:02:34.144 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:34.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T22:02:34.146 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-335007449100 2024-09-15T22:02:34.146 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:34.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449100 2024-09-15T22:02:34.148 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.8 seq 335007449100 2024-09-15T22:02:34.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 335007449100' 2024-09-15T22:02:34.148 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T22:02:34.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449101 -lt 335007449100 2024-09-15T22:02:34.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:02:34.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:02:34.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:02:35.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:02:35.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:02:35.202 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:02:35.202 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:02:35.202 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:02:35.202 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:02:35.203 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:02:35.203 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:02:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:02:35.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:02:35.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:02:35.603 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:02:36.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:02:36.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:02:36.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:02:36.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-15T22:02:36.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:02:36.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:02:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-15T22:02:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:02:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:02:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-15T22:02:36.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:02:36.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:02:36.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:465: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:02:36.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:02:36.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:02:36.709 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:02:36.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:02:37.254 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:02:37.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:02:37.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:02:37.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:02:37.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:02:37.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:02:37.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:02:37.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:02:37.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:02:37.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:02:37.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:02:37.583 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:02:37.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:02:37.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:02:37.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:02:37.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:02:37.587 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:02:37.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:02:37.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:02:37.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:02:37.589 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:02:37.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:02:37.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:02:37.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:02:37.642 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:02:37.643 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:37.643 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:37.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:02:37.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:02:37.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:02:37.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T22:02:37.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T22:02:37.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T22:02:37.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T22:02:37.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:02:37.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:02:37.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:02:37.645 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:02:37.645 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:02:37.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:02:37.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:02:37.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:02:37.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:02:37.648 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:02:37.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:02:37.649 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:02:37.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:02:37.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:02:37.650 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:02:37.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:02:37.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:02:37.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:02:37.652 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:02:37.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:02:37.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:02:37.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:02:37.655 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:02:37.655 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:37.655 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:37.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:02:37.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:02:37.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:02:37.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T22:02:37.658 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:02:37.658 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:37.658 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:37.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T22:02:37.660 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:02:37.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:02:37.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:02:37.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T22:02:37.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T22:02:37.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T22:02:37.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:02:37.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:02:37.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:02:37.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T22:02:37.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T22:02:38.130 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:02:38.130 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:02:38.131 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:02:38.131 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:02:38.131 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:38.131 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:38.131 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:02:38.131 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T22:02:38.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:02:38.340 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:02:38.340 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:02:38.340 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:02:38.340 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:02:38.341 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:02:38.341 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:02:38.341 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:02:38.341 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:02:38.341 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:38.341 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:38.342 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:38.342 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:02:38.342 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:02:38.342 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T22:02:38.459 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:02:38.459 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:02:38.459 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:02:38.459 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:02:38.459 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:02:38.460 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:02:38.460 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:02:38.460 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:02:38.461 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:38.461 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:38.461 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:38.461 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:02:38.462 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:02:38.462 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T22:02:38.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T22:02:38.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T22:02:38.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:02:38.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:02:38.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:02:38.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T22:02:38.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:02:39.464 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:02:39.465 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:02:39.465 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:02:39.465 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:02:39.465 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:39.465 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:39.466 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:02:39.466 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:02:39.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:02:39.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T22:02:39.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T22:02:39.785 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T22:02:39.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:02:40.804 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T22:02:40.804 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:02:40.804 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:02:40.805 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:40.805 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:40.805 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:40.806 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:02:40.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T22:02:40.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T22:02:40.907 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:02:40.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T22:02:40.922 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:02:38.172+0000 7efce83c1e00 0 load: jerasure load: lrc 2024-09-15T22:02:40.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2024-09-15T22:02:40.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2024-09-15T22:02:40.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: local objname=myobject 2024-09-15T22:02:40.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:474: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:02:40.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:02:40.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:476: TEST_ec_recovery_multiple_objects: setup_osds 7 2024-09-15T22:02:40.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-15T22:02:40.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T22:02:40.924 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-15T22:02:40.926 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-15T22:02:40.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:02:40.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T22:02:40.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:02:40.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:02:40.929 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:02:40.930 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:02:40.930 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:02:40.930 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:40.930 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:02:40.930 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:02:40.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:02:40.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:02:40.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2d760ef0-139a-4cf0-8c6e-114d237077e1 2024-09-15T22:02:40.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2d760ef0-139a-4cf0-8c6e-114d237077e1' 2024-09-15T22:02:40.933 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 2d760ef0-139a-4cf0-8c6e-114d237077e1 2024-09-15T22:02:40.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:02:40.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCAWedmCE0dORAAlsznUBEd/c945gvxNRK9JQ== 2024-09-15T22:02:40.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCAWedmCE0dORAAlsznUBEd/c945gvxNRK9JQ=="}' 2024-09-15T22:02:40.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2d760ef0-139a-4cf0-8c6e-114d237077e1 -i td/test-erasure-eio/0/new.json 2024-09-15T22:02:41.246 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:02:41.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T22:02:41.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCAWedmCE0dORAAlsznUBEd/c945gvxNRK9JQ== --osd-uuid 2d760ef0-139a-4cf0-8c6e-114d237077e1 2024-09-15T22:02:41.294 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:02:41.292+0000 7feda95a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:41.313 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:02:41.308+0000 7feda95a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:41.315 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:02:41.312+0000 7feda95a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:41.315 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:02:41.312+0000 7feda95a87c0 -1 bdev(0x556223239c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:02:41.315 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:02:41.312+0000 7feda95a87c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T22:02:46.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T22:02:46.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:02:46.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:02:46.658 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T22:02:46.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:02:47.257 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T22:02:47.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:02:47.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:02:47.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:02:47.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:02:47.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:02:47.294 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:02:47.288+0000 7fec3ff947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:47.296 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:02:47.292+0000 7fec3ff947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:47.297 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:02:47.292+0000 7fec3ff947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:47.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:02:47.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:02:47.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:02:47.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:02:47.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:02:47.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:47.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:02:47.710 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:02:47.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:47.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:48.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:48.529 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:02:48.524+0000 7fec3ff947c0 -1 Falling back to public interface 2024-09-15T22:02:49.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:49.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:49.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:02:49.192 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:02:49.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:49.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:49.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:50.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:50.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:50.613 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:02:50.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:02:50.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:50.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:51.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:51.401 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:02:51.396+0000 7fec3ff947c0 -1 osd.0 0 log_to_monitors true 2024-09-15T22:02:52.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:52.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:52.043 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:02:52.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:02:52.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:52.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:52.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:53.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:53.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:53.731 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:02:53.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:02:53.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:53.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:54.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:55.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:55.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:55.334 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:02:55.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:02:55.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:55.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:55.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:56.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:56.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:56.795 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:02:56.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:02:56.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:56.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:57.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:58.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:58.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:58.292 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:02:58.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:02:58.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:58.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:58.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:59.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:59.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:59.760 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:02:59.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:02:59.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:59.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:00.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:01.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:01.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:01.272 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:03:01.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:03:01.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:01.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:01.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:02.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:02.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:02.844 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:03:02.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:03:02.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:02.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:03.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:04.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:04.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:04.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:03:04.651 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:03:04.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:04.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:05.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:06.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:06.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:06.556 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:03:06.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:03:06.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:06.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:07.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:08.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:08.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:08.376 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:03:08.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:03:08.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:08.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:08.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:09.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:09.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:09.827 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:03:09.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:03:09.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:09.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:10.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:11.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:11.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:11.615 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:03:11.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:03:11.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:11.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:12.567 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:13.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:13.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:13.569 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:03:13.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:03:13.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:13.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:14.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:15.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:15.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:15.139 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:03:15.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:03:15.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:15.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:15.680 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/469132356,v1:127.0.0.1:6803/469132356] [v2:127.0.0.1:6804/469132356,v1:127.0.0.1:6805/469132356] exists,up 2d760ef0-139a-4cf0-8c6e-114d237077e1 2024-09-15T22:03:15.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:03:15.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:03:15.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:03:15.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:03:15.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T22:03:15.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:03:15.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:03:15.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:03:15.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:03:15.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:03:15.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:03:15.686 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:03:15.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:03:15.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:03:15.688 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:03:15.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d9e2e968-b65e-467e-9d64-847bbbf46f35 2024-09-15T22:03:15.688 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 d9e2e968-b65e-467e-9d64-847bbbf46f35 2024-09-15T22:03:15.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d9e2e968-b65e-467e-9d64-847bbbf46f35' 2024-09-15T22:03:15.689 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:03:15.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCjWedmxgSVKhAA+fWuzK9wKowNLxBFNanipw== 2024-09-15T22:03:15.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCjWedmxgSVKhAA+fWuzK9wKowNLxBFNanipw=="}' 2024-09-15T22:03:15.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d9e2e968-b65e-467e-9d64-847bbbf46f35 -i td/test-erasure-eio/1/new.json 2024-09-15T22:03:16.281 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:03:16.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T22:03:16.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjWedmxgSVKhAA+fWuzK9wKowNLxBFNanipw== --osd-uuid d9e2e968-b65e-467e-9d64-847bbbf46f35 2024-09-15T22:03:16.342 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:16.340+0000 7f51cfeb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:16.348 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:16.344+0000 7f51cfeb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:16.349 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:16.344+0000 7f51cfeb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:16.350 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:16.344+0000 7f51cfeb27c0 -1 bdev(0x556d3d803c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:03:16.350 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:16.348+0000 7f51cfeb27c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T22:03:22.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T22:03:22.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:03:22.296 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T22:03:22.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:03:22.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:03:22.879 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T22:03:22.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:03:22.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:03:22.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:03:22.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:03:22.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:03:22.917 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:22.912+0000 7fa8992697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:22.946 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:22.944+0000 7fa8992697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:22.948 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:22.944+0000 7fa8992697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:23.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:03:23.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:03:23.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:03:23.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:03:23.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:03:23.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:23.377 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:03:23.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:03:23.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:23.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:23.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:24.005 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:24.000+0000 7fa8992697c0 -1 Falling back to public interface 2024-09-15T22:03:24.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:24.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:24.801 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:03:24.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:03:24.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:24.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:25.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:26.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:26.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:26.288 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:03:26.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:03:26.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:26.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:26.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:26.844 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:26.840+0000 7fa8992697c0 -1 osd.1 0 log_to_monitors true 2024-09-15T22:03:27.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:27.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:27.772 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:03:27.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:03:27.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:27.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:28.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:29.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:29.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:29.236 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:03:29.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:03:29.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:29.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:29.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:30.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:30.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:30.747 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:03:30.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:03:30.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:30.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:31.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:32.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:32.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:32.197 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:03:32.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:03:32.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:32.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:32.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:33.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:33.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:33.824 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:03:33.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:03:33.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:33.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:34.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:35.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:35.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:35.305 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:03:35.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:03:35.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:35.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:36.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:36.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:36.827 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:03:36.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:03:36.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:36.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:37.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:38.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:38.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:38.304 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:03:38.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:03:38.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:38.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:38.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:39.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:39.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:39.918 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:03:39.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:03:39.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:39.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:40.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:41.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:41.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:41.564 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:03:41.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:03:41.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:41.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:42.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:43.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:43.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:43.360 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:03:43.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:03:43.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:43.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:44.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:45.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:45.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:45.165 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:03:45.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:03:45.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:45.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:45.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:46.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:46.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:46.914 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:03:46.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:03:46.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:46.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:47.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:48.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:48.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:48.994 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:03:48.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:03:48.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:48.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:49.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:50.935 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:50.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:50.936 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:03:50.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:03:50.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:50.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:51.408 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2750561067,v1:127.0.0.1:6811/2750561067] [v2:127.0.0.1:6812/2750561067,v1:127.0.0.1:6813/2750561067] exists,up d9e2e968-b65e-467e-9d64-847bbbf46f35 2024-09-15T22:03:51.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:03:51.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:03:51.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:03:51.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:03:51.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:03:51.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:03:51.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:03:51.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:03:51.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:03:51.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:03:51.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:03:51.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:03:51.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:03:51.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:03:51.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:03:51.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:03:51.415 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:03:51.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a4042642-70c5-4dee-a07a-3b369324ce13 2024-09-15T22:03:51.416 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 a4042642-70c5-4dee-a07a-3b369324ce13 2024-09-15T22:03:51.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 a4042642-70c5-4dee-a07a-3b369324ce13' 2024-09-15T22:03:51.417 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:03:51.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDHWedm6FFdGhAAlBKTIz4lA28/popjxBdf4g== 2024-09-15T22:03:51.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDHWedm6FFdGhAAlBKTIz4lA28/popjxBdf4g=="}' 2024-09-15T22:03:51.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a4042642-70c5-4dee-a07a-3b369324ce13 -i td/test-erasure-eio/2/new.json 2024-09-15T22:03:51.967 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:03:51.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T22:03:51.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDHWedm6FFdGhAAlBKTIz4lA28/popjxBdf4g== --osd-uuid a4042642-70c5-4dee-a07a-3b369324ce13 2024-09-15T22:03:52.019 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:52.020+0000 7f11f721f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:52.103 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:52.104+0000 7f11f721f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:52.104 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:52.104+0000 7f11f721f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:52.105 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:52.104+0000 7f11f721f7c0 -1 bdev(0x557e17726800 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:03:52.105 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:52.104+0000 7f11f721f7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T22:03:58.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T22:03:58.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:03:58.291 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T22:03:58.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:03:58.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:03:58.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:03:58.874 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:03:58.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:03:58.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:03:58.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:03:58.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:03:58.910 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:58.907+0000 7f2ce097b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:58.912 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:58.911+0000 7f2ce097b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:58.913 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:58.911+0000 7f2ce097b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:59.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:03:59.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:03:59.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:03:59.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:03:59.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:03:59.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:59.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:03:59.354 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:03:59.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:59.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:03:59.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:59.949 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:03:59.947+0000 7f2ce097b7c0 -1 Falling back to public interface 2024-09-15T22:04:00.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:00.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:00.800 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:04:00.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:04:00.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:00.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:01.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:02.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:02.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:02.285 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:04:02.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:04:02.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:02.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:02.665 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:04:02.663+0000 7f2ce097b7c0 -1 osd.2 0 log_to_monitors true 2024-09-15T22:04:02.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:03.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:03.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:03.774 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:04:03.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:04:03.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:03.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:04.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:05.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:05.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:05.223 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:04:05.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:04:05.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:05.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:05.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:06.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:06.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:06.674 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:04:06.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:04:06.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:06.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:07.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:08.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:08.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:08.285 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:04:08.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:04:08.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:08.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:08.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:09.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:09.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:09.770 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:04:09.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:04:09.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:09.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:10.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:11.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:11.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:11.228 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:04:11.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:04:11.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:11.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:11.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:12.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:12.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:12.698 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:04:12.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:04:12.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:12.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:13.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:14.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:14.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:14.126 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:04:14.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:04:14.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:14.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:14.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:15.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:15.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:15.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:04:15.772 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:04:15.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:15.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:16.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:17.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:17.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:17.227 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:04:17.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:04:17.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:17.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:18.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:19.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:19.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:19.046 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:04:19.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:04:19.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:19.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:20.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:21.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:21.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:21.257 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:04:21.257 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:04:21.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:21.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:21.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:22.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:22.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:22.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:04:22.984 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:04:22.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:22.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:24.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:25.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:25.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:25.179 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:04:25.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:04:25.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:25.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:26.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:27.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:27.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:27.342 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:04:27.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:04:27.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:27.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:04:27.974 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1938695746,v1:127.0.0.1:6819/1938695746] [v2:127.0.0.1:6820/1938695746,v1:127.0.0.1:6821/1938695746] exists,up a4042642-70c5-4dee-a07a-3b369324ce13 2024-09-15T22:04:27.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:04:27.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:04:27.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:04:27.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:04:27.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T22:04:27.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:04:27.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:04:27.978 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:04:27.979 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:04:27.979 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:04:27.979 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:04:27.979 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:04:27.979 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:04:27.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:04:27.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7c6f5eb7-e48c-4469-8951-68c9b99bc24a 2024-09-15T22:04:27.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 7c6f5eb7-e48c-4469-8951-68c9b99bc24a' 2024-09-15T22:04:27.982 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 7c6f5eb7-e48c-4469-8951-68c9b99bc24a 2024-09-15T22:04:27.982 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:04:28.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDsWedmC28aABAAkV0Yy4KdMLuT8kr6f26Vvw== 2024-09-15T22:04:28.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDsWedmC28aABAAkV0Yy4KdMLuT8kr6f26Vvw=="}' 2024-09-15T22:04:28.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7c6f5eb7-e48c-4469-8951-68c9b99bc24a -i td/test-erasure-eio/3/new.json 2024-09-15T22:04:28.841 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:04:28.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T22:04:28.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsWedmC28aABAAkV0Yy4KdMLuT8kr6f26Vvw== --osd-uuid 7c6f5eb7-e48c-4469-8951-68c9b99bc24a 2024-09-15T22:04:28.900 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:04:28.899+0000 7ff2353b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:28.902 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:04:28.903+0000 7ff2353b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:28.903 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:04:28.903+0000 7ff2353b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:28.904 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:04:28.903+0000 7ff2353b37c0 -1 bdev(0x557dba5b3c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T22:04:28.904 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:04:28.903+0000 7ff2353b37c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T22:04:35.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T22:04:35.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:04:35.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T22:04:35.281 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T22:04:35.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:04:35.865 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T22:04:35.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T22:04:35.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:04:35.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:04:35.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:04:35.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:04:35.897 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:04:35.895+0000 7fba5038f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:35.916 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:04:35.915+0000 7fba5038f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:35.917 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:04:35.915+0000 7fba5038f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:36.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T22:04:36.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:04:36.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:04:36.353 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:04:36.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:04:36.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:04:36.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:36.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:04:36.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:36.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:36.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:37.181 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:04:37.179+0000 7fba5038f7c0 -1 Falling back to public interface 2024-09-15T22:04:37.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:37.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:37.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:04:37.797 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:04:37.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:37.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:38.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:39.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:39.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:39.220 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:04:39.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:04:39.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:39.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:39.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:40.003 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:04:40.003+0000 7fba5038f7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T22:04:40.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:40.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:40.658 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:04:40.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:04:40.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:40.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:41.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:42.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:42.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:42.358 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:04:42.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:04:42.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:42.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:43.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:44.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:44.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:44.012 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:04:44.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:04:44.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:44.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:44.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:45.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:45.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:45.476 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:04:45.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:04:45.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:45.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:45.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:46.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:46.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:46.956 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:04:46.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:04:46.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:46.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:47.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:48.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:48.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:48.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:04:48.464 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:04:48.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:48.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:48.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:49.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:49.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:49.910 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:04:49.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:04:49.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:49.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:50.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:51.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:51.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:51.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:04:51.335 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:04:51.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:51.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:51.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:52.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:52.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:52.922 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:04:52.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:04:52.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:52.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:53.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:54.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:54.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:54.582 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:04:54.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:04:54.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:54.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:55.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:56.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:56.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:56.739 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:04:56.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:04:56.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:56.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:57.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:58.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:58.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:58.244 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:04:58.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:04:58.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:58.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:04:58.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:59.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:59.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:59.738 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:04:59.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:04:59.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:59.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:05:00.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:01.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:01.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:01.669 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:05:01.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:05:01.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:01.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:05:02.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:03.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:03.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:03.520 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:05:03.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:05:03.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:03.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:05:04.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:05.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:05.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:05.039 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:05:05.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:05:05.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:05.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:05:05.603 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/797063757,v1:127.0.0.1:6827/797063757] [v2:127.0.0.1:6828/797063757,v1:127.0.0.1:6829/797063757] exists,up 7c6f5eb7-e48c-4469-8951-68c9b99bc24a 2024-09-15T22:05:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:05:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:05:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:05:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:05:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T22:05:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:05:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:05:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-15T22:05:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:05:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T22:05:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:05:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:05:05.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T22:05:05.609 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:05:05.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3cef63b3-0c80-4673-ae3f-948c31fea427 2024-09-15T22:05:05.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 3cef63b3-0c80-4673-ae3f-948c31fea427' 2024-09-15T22:05:05.610 INFO:tasks.workunit.client.0.smithi104.stdout:add osd4 3cef63b3-0c80-4673-ae3f-948c31fea427 2024-09-15T22:05:05.610 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:05:05.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQARWudmwZXiJRAArhZYBk/yokuyGsMh4LqclQ== 2024-09-15T22:05:05.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQARWudmwZXiJRAArhZYBk/yokuyGsMh4LqclQ=="}' 2024-09-15T22:05:05.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3cef63b3-0c80-4673-ae3f-948c31fea427 -i td/test-erasure-eio/4/new.json 2024-09-15T22:05:06.374 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:05:06.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T22:05:06.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARWudmwZXiJRAArhZYBk/yokuyGsMh4LqclQ== --osd-uuid 3cef63b3-0c80-4673-ae3f-948c31fea427 2024-09-15T22:05:06.431 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:06.431+0000 7f504c0327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:06.589 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:06.587+0000 7f504c0327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:06.591 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:06.591+0000 7f504c0327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:06.591 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:06.591+0000 7f504c0327c0 -1 bdev(0x56518b0b7c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-15T22:05:06.591 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:06.591+0000 7f504c0327c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T22:05:13.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T22:05:13.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:05:13.376 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd4 key to auth repository 2024-09-15T22:05:13.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-15T22:05:13.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:05:13.959 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.4 2024-09-15T22:05:13.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-15T22:05:13.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:05:13.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:05:13.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:05:13.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:05:13.995 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:13.995+0000 7f7eff2d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:14.022 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:14.019+0000 7f7eff2d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:14.023 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:14.023+0000 7f7eff2d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:14.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-15T22:05:14.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:05:14.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-15T22:05:14.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:05:14.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:05:14.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:14.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:05:14.454 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:05:14.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:14.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:14.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:15.269 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:15.267+0000 7f7eff2d07c0 -1 Falling back to public interface 2024-09-15T22:05:15.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:15.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:15.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:05:15.904 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:05:15.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:15.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:16.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:17.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:17.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:17.361 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:05:17.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:05:17.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:17.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:17.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:18.113 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:18.111+0000 7f7eff2d07c0 -1 osd.4 0 log_to_monitors true 2024-09-15T22:05:18.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:18.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:18.804 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:05:18.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:05:18.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:18.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:19.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:20.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:20.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:20.559 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:05:20.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:05:20.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:20.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:21.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:22.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:22.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:22.262 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:05:22.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:05:22.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:22.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:22.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:23.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:23.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:23.877 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:05:23.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:05:23.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:23.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:24.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:25.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:25.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:25.327 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:05:25.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:05:25.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:25.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:25.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:26.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:26.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:26.778 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:05:26.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:05:26.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:26.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:27.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:28.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:28.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:28.336 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:05:28.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:05:28.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:28.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:28.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:29.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:29.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:29.877 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:05:29.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:05:29.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:29.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:30.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:31.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:31.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:31.355 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:05:31.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:05:31.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:31.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:31.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:32.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:32.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:32.802 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:05:32.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:05:32.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:32.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:33.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:34.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:34.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:34.419 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:05:34.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:05:34.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:34.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:34.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:35.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:35.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:35.887 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:05:35.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:05:35.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:35.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:36.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:37.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:37.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:37.640 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:05:37.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:05:37.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:37.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:38.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:39.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:39.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:39.367 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:05:39.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:05:39.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:39.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:40.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:41.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:41.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:41.111 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:05:41.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:05:41.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:41.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:41.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:42.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:42.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:42.913 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:05:42.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:05:42.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:42.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:05:43.365 INFO:tasks.workunit.client.0.smithi104.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2917184189,v1:127.0.0.1:6835/2917184189] [v2:127.0.0.1:6836/2917184189,v1:127.0.0.1:6837/2917184189] exists,up 3cef63b3-0c80-4673-ae3f-948c31fea427 2024-09-15T22:05:43.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:05:43.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:05:43.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:05:43.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:05:43.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-15T22:05:43.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:05:43.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:05:43.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-15T22:05:43.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:05:43.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-15T22:05:43.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:05:43.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:05:43.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:05:43.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:05:43.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-15T22:05:43.372 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:05:43.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f9ad214e-2f93-46dd-beb5-d95b3954b87a 2024-09-15T22:05:43.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 f9ad214e-2f93-46dd-beb5-d95b3954b87a' 2024-09-15T22:05:43.372 INFO:tasks.workunit.client.0.smithi104.stdout:add osd5 f9ad214e-2f93-46dd-beb5-d95b3954b87a 2024-09-15T22:05:43.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:05:43.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA3WudmwcKnFxAArcP0AJnFpV74MfqD/sBkOw== 2024-09-15T22:05:43.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA3WudmwcKnFxAArcP0AJnFpV74MfqD/sBkOw=="}' 2024-09-15T22:05:43.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f9ad214e-2f93-46dd-beb5-d95b3954b87a -i td/test-erasure-eio/5/new.json 2024-09-15T22:05:44.321 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:05:44.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-15T22:05:44.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3WudmwcKnFxAArcP0AJnFpV74MfqD/sBkOw== --osd-uuid f9ad214e-2f93-46dd-beb5-d95b3954b87a 2024-09-15T22:05:44.381 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:44.378+0000 7f9a19c0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:44.463 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:44.462+0000 7f9a19c0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:44.465 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:44.462+0000 7f9a19c0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:44.465 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:44.462+0000 7f9a19c0f7c0 -1 bdev(0x555590993c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-15T22:05:44.465 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:44.462+0000 7f9a19c0f7c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-15T22:05:52.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-15T22:05:52.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:05:52.017 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd5 key to auth repository 2024-09-15T22:05:52.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-15T22:05:52.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:05:52.617 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.5 2024-09-15T22:05:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-15T22:05:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:05:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:05:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:05:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:05:52.654 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:52.654+0000 7f497d9917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:52.671 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:52.670+0000 7f497d9917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:52.672 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:52.670+0000 7f497d9917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:53.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-15T22:05:53.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:05:53.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T22:05:53.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:05:53.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:05:53.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:53.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:05:53.062 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:05:53.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:53.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:05:53.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:54.153 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:54.150+0000 7f497d9917c0 -1 Falling back to public interface 2024-09-15T22:05:54.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:54.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:54.478 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:05:54.479 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:05:54.479 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:54.479 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:05:54.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:55.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:55.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:55.920 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:05:55.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:05:55.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:55.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:05:56.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:57.082 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:05:57.078+0000 7f497d9917c0 -1 osd.5 0 log_to_monitors true 2024-09-15T22:05:57.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:57.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:57.404 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:05:57.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:05:57.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:57.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:05:57.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:58.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:58.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:58.959 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:05:58.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:05:58.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:58.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:05:59.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:00.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:00.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:00.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:06:00.482 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:06:00.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:00.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:01.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:02.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:02.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:02.013 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:06:02.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:06:02.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:02.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:02.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:03.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:03.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:03.833 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:06:03.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:06:03.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:03.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:04.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:05.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:05.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:05.345 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:06:05.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:06:05.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:05.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:05.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:06.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:06.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:06.790 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:06:06.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:06:06.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:06.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:07.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:08.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:08.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:08.318 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:06:08.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:06:08.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:08.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:08.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:09.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:09.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:09.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:06:09.906 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:06:09.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:09.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:10.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:11.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:11.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:11.353 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:06:11.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:06:11.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:11.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:12.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:13.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:13.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:13.152 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:06:13.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:06:13.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:13.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:14.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:15.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:15.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:15.032 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:06:15.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:06:15.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:15.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:15.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:16.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:16.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:16.743 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:06:16.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:06:16.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:16.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:17.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:18.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:18.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:18.965 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:06:18.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:06:18.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:18.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:20.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:21.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:21.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:21.032 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:06:21.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:06:21.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:21.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:06:21.518 INFO:tasks.workunit.client.0.smithi104.stdout:osd.5 up in weight 1 up_from 47 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2372492510,v1:127.0.0.1:6843/2372492510] [v2:127.0.0.1:6844/2372492510,v1:127.0.0.1:6845/2372492510] exists,up f9ad214e-2f93-46dd-beb5-d95b3954b87a 2024-09-15T22:06:21.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:06:21.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:06:21.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:06:21.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-15T22:06:21.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:06:21.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:06:21.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:06:21.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:06:21.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:06:21.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:06:21.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:06:21.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:06:21.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:06:21.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:06:21.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:06:21.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-15T22:06:21.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:06:21.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=568b0653-4101-4171-9f29-bd3c1d5d7988 2024-09-15T22:06:21.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 568b0653-4101-4171-9f29-bd3c1d5d7988' 2024-09-15T22:06:21.525 INFO:tasks.workunit.client.0.smithi104.stdout:add osd6 568b0653-4101-4171-9f29-bd3c1d5d7988 2024-09-15T22:06:21.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:06:21.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBdWudmBdHBIBAAq3WyJer2/csgNfhpNEzs+g== 2024-09-15T22:06:21.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBdWudmBdHBIBAAq3WyJer2/csgNfhpNEzs+g=="}' 2024-09-15T22:06:21.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 568b0653-4101-4171-9f29-bd3c1d5d7988 -i td/test-erasure-eio/6/new.json 2024-09-15T22:06:22.211 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:06:22.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-15T22:06:22.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdWudmBdHBIBAAq3WyJer2/csgNfhpNEzs+g== --osd-uuid 568b0653-4101-4171-9f29-bd3c1d5d7988 2024-09-15T22:06:22.266 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:06:22.262+0000 7f81aa4d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:06:22.309 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:06:22.306+0000 7f81aa4d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:06:22.311 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:06:22.310+0000 7f81aa4d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:06:22.312 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:06:22.310+0000 7f81aa4d27c0 -1 bdev(0x55cc1ed03c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-15T22:06:22.312 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:06:22.310+0000 7f81aa4d27c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-15T22:06:29.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-15T22:06:29.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:06:29.019 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd6 key to auth repository 2024-09-15T22:06:29.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-15T22:06:29.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:06:29.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-15T22:06:29.607 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.6 2024-09-15T22:06:29.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:06:29.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:06:29.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:06:29.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:06:29.643 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:06:29.642+0000 7fdf3a87f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:06:29.651 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:06:29.650+0000 7fdf3a87f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:06:29.652 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:06:29.650+0000 7fdf3a87f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:06:30.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-15T22:06:30.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:06:30.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-15T22:06:30.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:06:30.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:06:30.028 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:06:30.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:30.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:06:30.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:30.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:30.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:31.185 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:06:31.182+0000 7fdf3a87f7c0 -1 Falling back to public interface 2024-09-15T22:06:31.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:31.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:31.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:06:31.463 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:06:31.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:31.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:31.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:32.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:32.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:32.943 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:06:32.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:06:32.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:32.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:33.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:33.986 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:06:33.986+0000 7fdf3a87f7c0 -1 osd.6 0 log_to_monitors true 2024-09-15T22:06:34.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:34.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:34.425 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:06:34.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:06:34.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:34.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:35.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:36.113 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:06:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:06:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:36.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:37.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:37.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:37.795 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:06:37.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:06:37.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:37.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:38.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:39.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:39.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:39.224 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:06:39.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:06:39.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:39.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:39.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:40.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:40.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:40.790 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:06:40.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:06:40.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:40.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:41.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:42.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:42.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:42.248 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:06:42.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:06:42.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:42.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:42.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:43.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:43.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:43.777 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:06:43.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:06:43.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:43.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:44.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:45.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:45.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:45.229 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:06:45.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:06:45.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:45.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:45.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:46.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:46.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:46.820 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:06:46.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:06:46.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:46.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:47.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:48.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:48.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:48.291 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:06:48.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:06:48.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:48.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:49.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:50.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:50.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:50.113 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:06:50.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:06:50.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:50.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:50.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:51.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:51.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:51.856 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:06:51.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:06:51.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:51.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:52.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:53.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:53.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:53.606 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:06:53.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:06:53.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:53.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:54.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:55.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:55.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:55.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:06:55.388 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:06:55.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:55.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:56.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:57.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:57.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:57.180 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:06:57.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:06:57.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:57.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:58.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:06:59.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:06:59.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:06:59.264 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:06:59.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:06:59.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:06:59.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:06:59.719 INFO:tasks.workunit.client.0.smithi104.stdout:osd.6 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3508342086,v1:127.0.0.1:6851/3508342086] [v2:127.0.0.1:6852/3508342086,v1:127.0.0.1:6853/3508342086] exists,up 568b0653-4101-4171-9f29-bd3c1d5d7988 2024-09-15T22:06:59.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:06:59.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:06:59.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:06:59.720 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:06:59.720 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:06:59.720 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:06:59.721 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:06:59.721 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:06:59.721 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:06:59.721 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:06:59.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:06:59.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:06:59.829 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:06:59.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:06:59.850 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:02:48.536+0000 7fec3ff947c0 0 load: jerasure load: lrc 2024-09-15T22:06:59.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:477: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:06:59.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:479: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2024-09-15T22:06:59.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T22:06:59.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:06:59.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:06:59.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T22:06:59.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:06:59.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T22:06:59.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:06:59.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T22:07:00.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:07:00.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:07:01.812 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:07:01.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:07:02.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:07:02.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:07:02.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:07:02.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:07:02.835 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:07:02.836 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:07:02.836 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:07:02.837 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:07:02.837 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:07:02.837 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:07:02.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:07:02.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:07:02.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:07:02.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:07:02.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:07:02.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:07:03.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:07:03.551 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:07:03.551 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:07:03.551 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:07:03.551 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:07:03.551 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:07:03.551 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T22:07:03.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:07:03.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:03.552 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:07:03.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803827 2024-09-15T22:07:03.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803827 2024-09-15T22:07:03.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827' 2024-09-15T22:07:03.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:03.782 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:07:03.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574897 2024-09-15T22:07:03.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574897 2024-09-15T22:07:03.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574897' 2024-09-15T22:07:03.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:03.962 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:07:04.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378664 2024-09-15T22:07:04.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378664 2024-09-15T22:07:04.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574897 2-81604378664' 2024-09-15T22:07:04.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:04.115 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:07:04.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084321 2024-09-15T22:07:04.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084321 2024-09-15T22:07:04.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574897 2-81604378664 3-120259084321' 2024-09-15T22:07:04.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:04.272 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T22:07:04.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757269 2024-09-15T22:07:04.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757269 2024-09-15T22:07:04.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574897 2-81604378664 3-120259084321 4-163208757269' 2024-09-15T22:07:04.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:04.425 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T22:07:04.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462926 2024-09-15T22:07:04.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462926 2024-09-15T22:07:04.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574897 2-81604378664 3-120259084321 4-163208757269 5-201863462926' 2024-09-15T22:07:04.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:04.583 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T22:07:04.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201290 2024-09-15T22:07:04.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201290 2024-09-15T22:07:04.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574897 2-81604378664 3-120259084321 4-163208757269 5-201863462926 6-236223201290' 2024-09-15T22:07:04.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:04.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803827 2024-09-15T22:07:04.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:04.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:07:04.739 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803827 2024-09-15T22:07:04.739 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:04.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803827 2024-09-15T22:07:04.740 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803827 2024-09-15T22:07:04.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-09-15T22:07:04.741 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:07:05.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803827 -lt 25769803827 2024-09-15T22:07:05.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:05.162 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574897 2024-09-15T22:07:05.162 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:05.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:07:05.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:05.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574897 2024-09-15T22:07:05.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574897 2024-09-15T22:07:05.165 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574897 2024-09-15T22:07:05.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574897' 2024-09-15T22:07:05.166 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:07:05.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574897 -lt 55834574897 2024-09-15T22:07:05.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:05.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378664 2024-09-15T22:07:05.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:05.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:07:05.636 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378664 2024-09-15T22:07:05.636 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:05.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378664 2024-09-15T22:07:05.639 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 81604378664 2024-09-15T22:07:05.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378664' 2024-09-15T22:07:05.639 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:07:06.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378664 -lt 81604378664 2024-09-15T22:07:06.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:06.067 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084321 2024-09-15T22:07:06.067 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:06.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:07:06.069 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084321 2024-09-15T22:07:06.070 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:06.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084321 2024-09-15T22:07:06.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084321' 2024-09-15T22:07:06.071 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 120259084321 2024-09-15T22:07:06.071 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:07:06.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084322 -lt 120259084321 2024-09-15T22:07:06.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:06.508 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757269 2024-09-15T22:07:06.508 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:06.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T22:07:06.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:06.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757269 2024-09-15T22:07:06.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757269 2024-09-15T22:07:06.512 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 163208757269 2024-09-15T22:07:06.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757269' 2024-09-15T22:07:06.512 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T22:07:06.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757269 -lt 163208757269 2024-09-15T22:07:06.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:06.988 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-201863462926 2024-09-15T22:07:06.989 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:06.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T22:07:06.991 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-201863462926 2024-09-15T22:07:06.991 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:06.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462926 2024-09-15T22:07:06.993 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 201863462926 2024-09-15T22:07:06.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 201863462926' 2024-09-15T22:07:06.994 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T22:07:07.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462926 -lt 201863462926 2024-09-15T22:07:07.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:07.419 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:07.419 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-236223201290 2024-09-15T22:07:07.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T22:07:07.420 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-236223201290 2024-09-15T22:07:07.421 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:07.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201290 2024-09-15T22:07:07.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 236223201290' 2024-09-15T22:07:07.422 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 236223201290 2024-09-15T22:07:07.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T22:07:08.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201290 -lt 236223201290 2024-09-15T22:07:08.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:07:08.030 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:07:08.030 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:07:08.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:07:08.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:07:08.590 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:07:08.590 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:07:08.590 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:07:08.591 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:07:08.591 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:07:08.591 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:07:09.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:07:09.024 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:07:09.024 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:07:09.024 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:07:09.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:07:09.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:07:09.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:07:09.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:482: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-15T22:07:09.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:07:09.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:07:09.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-15T22:07:09.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:09.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:07:09.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:09.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:07:09.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:09.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:07:09.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:09.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:07:09.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-15T22:07:09.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-15T22:07:09.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:07:09.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:07:09.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-15T22:07:09.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:09.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:07:09.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:09.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:07:09.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:09.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:07:09.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:09.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:07:09.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-15T22:07:10.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-15T22:07:10.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:07:10.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:07:10.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-15T22:07:10.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:10.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:07:10.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:10.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:07:10.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:10.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:07:10.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:07:10.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:07:10.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-15T22:07:10.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects: ceph osd out 0 2024-09-15T22:07:10.868 INFO:tasks.workunit.client.0.smithi104.stderr:osd.0 is already out. 2024-09-15T22:07:10.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects: wait_for_clean 2024-09-15T22:07:10.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:07:10.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:07:10.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:07:10.884 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:07:10.884 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:07:10.884 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:07:10.885 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:07:10.885 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:07:10.885 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:07:10.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:07:10.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:07:10.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:07:10.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:07:10.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:07:10.985 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:07:11.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:07:11.608 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:07:11.608 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:07:11.608 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:07:11.609 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:07:11.609 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:07:11.609 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T22:07:11.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:07:11.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:11.609 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:07:11.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803831 2024-09-15T22:07:11.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803831 2024-09-15T22:07:11.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831' 2024-09-15T22:07:11.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:11.782 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:07:12.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574901 2024-09-15T22:07:12.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574901 2024-09-15T22:07:12.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574901' 2024-09-15T22:07:12.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:12.034 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:07:12.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378667 2024-09-15T22:07:12.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378667 2024-09-15T22:07:12.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574901 2-81604378667' 2024-09-15T22:07:12.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:12.186 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:07:12.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084325 2024-09-15T22:07:12.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084325 2024-09-15T22:07:12.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574901 2-81604378667 3-120259084325' 2024-09-15T22:07:12.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:12.419 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T22:07:12.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757272 2024-09-15T22:07:12.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757272 2024-09-15T22:07:12.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574901 2-81604378667 3-120259084325 4-163208757272' 2024-09-15T22:07:12.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:12.575 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T22:07:12.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462930 2024-09-15T22:07:12.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462930 2024-09-15T22:07:12.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574901 2-81604378667 3-120259084325 4-163208757272 5-201863462930' 2024-09-15T22:07:12.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:07:12.725 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T22:07:12.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201293 2024-09-15T22:07:12.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201293 2024-09-15T22:07:12.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574901 2-81604378667 3-120259084325 4-163208757272 5-201863462930 6-236223201293' 2024-09-15T22:07:12.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:12.925 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:12.925 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803831 2024-09-15T22:07:12.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:07:12.927 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803831 2024-09-15T22:07:12.927 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:12.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803831 2024-09-15T22:07:12.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-09-15T22:07:12.929 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803831 2024-09-15T22:07:12.929 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:07:13.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803831 -lt 25769803831 2024-09-15T22:07:13.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:13.385 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574901 2024-09-15T22:07:13.385 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:13.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:07:13.387 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574901 2024-09-15T22:07:13.387 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:13.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574901 2024-09-15T22:07:13.389 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574901 2024-09-15T22:07:13.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574901' 2024-09-15T22:07:13.389 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:07:13.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574901 -lt 55834574901 2024-09-15T22:07:13.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:13.809 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378667 2024-09-15T22:07:13.809 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:13.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:07:13.811 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378667 2024-09-15T22:07:13.811 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:13.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378667 2024-09-15T22:07:13.812 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 81604378667 2024-09-15T22:07:13.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378667' 2024-09-15T22:07:13.813 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:07:14.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378667 -lt 81604378667 2024-09-15T22:07:14.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:14.295 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:14.295 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084325 2024-09-15T22:07:14.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:07:14.297 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084325 2024-09-15T22:07:14.297 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:14.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084325 2024-09-15T22:07:14.298 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 120259084325 2024-09-15T22:07:14.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084325' 2024-09-15T22:07:14.299 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:07:14.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084325 -lt 120259084325 2024-09-15T22:07:14.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:14.723 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757272 2024-09-15T22:07:14.723 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:14.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T22:07:14.726 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757272 2024-09-15T22:07:14.726 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:14.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757272 2024-09-15T22:07:14.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757272' 2024-09-15T22:07:14.727 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 163208757272 2024-09-15T22:07:14.728 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T22:07:15.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757273 -lt 163208757272 2024-09-15T22:07:15.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:15.160 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-201863462930 2024-09-15T22:07:15.161 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:15.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T22:07:15.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:15.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-201863462930 2024-09-15T22:07:15.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462930 2024-09-15T22:07:15.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 201863462930' 2024-09-15T22:07:15.165 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 201863462930 2024-09-15T22:07:15.165 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T22:07:15.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462930 -lt 201863462930 2024-09-15T22:07:15.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:07:15.650 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-236223201293 2024-09-15T22:07:15.650 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:07:15.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T22:07:15.652 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-236223201293 2024-09-15T22:07:15.652 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:07:15.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201293 2024-09-15T22:07:15.654 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 236223201293 2024-09-15T22:07:15.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 236223201293' 2024-09-15T22:07:15.654 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T22:07:16.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201293 -lt 236223201293 2024-09-15T22:07:16.085 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:07:16.086 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:07:16.086 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:07:16.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:07:16.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:07:16.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:07:16.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:07:16.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:07:16.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:07:16.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:07:16.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:07:17.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T22:07:17.045 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:07:17.046 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:07:17.046 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:07:17.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T22:07:17.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T22:07:17.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T22:07:17.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T22:07:17.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T22:07:17.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T22:07:17.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T22:07:17.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:07:17.684 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:07:17.684 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:07:17.684 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:07:17.684 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:07:17.685 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:07:17.685 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:07:18.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T22:07:18.096 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:07:18.097 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:07:18.097 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:07:18.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T22:07:18.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T22:07:18.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T22:07:18.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T22:07:18.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T22:07:18.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T22:07:18.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T22:07:18.650 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T22:07:18.650 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T22:07:19.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=3071 2024-09-15T22:07:19.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 3071 '!=' null 2024-09-15T22:07:19.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-15T22:07:19.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T22:07:19.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T22:07:19.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T22:07:19.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:07:19.295 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:07:19.296 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:07:19.296 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:07:19.296 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:07:19.296 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:07:19.296 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:07:19.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:07:19.706 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:07:19.707 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:07:19.707 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:07:20.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:07:20.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:07:20.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:07:20.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-15T22:07:20.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:07:20.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:07:20.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-15T22:07:20.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:07:20.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:07:20.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-15T22:07:20.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:07:20.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:07:20.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-15T22:07:20.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:07:20.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:07:20.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-15T22:07:20.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:07:20.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:07:20.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-15T22:07:20.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:07:20.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:07:20.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-15T22:07:20.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:07:20.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:07:20.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-15T22:07:20.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:07:20.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:07:20.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-15T22:07:20.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:07:20.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:07:20.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:495: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:07:20.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:07:20.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:07:20.994 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:07:21.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:07:21.669 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:07:21.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:07:21.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:07:21.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:07:21.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:07:21.688 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:07:21.688 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:07:21.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:07:21.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:07:21.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:07:21.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:07:21.998 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:07:21.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:07:22.000 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:07:22.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:07:22.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:07:22.003 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:07:22.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:07:22.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:07:22.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:07:22.004 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:07:22.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:07:22.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:07:22.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:07:22.048 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:07:22.048 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:07:22.048 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:07:22.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:07:22.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:07:22.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:07:22.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T22:07:22.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T22:07:22.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T22:07:22.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T22:07:22.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:07:22.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:07:22.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:07:22.051 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:07:22.051 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:07:22.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:07:22.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:07:22.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:07:22.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:07:22.053 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:07:22.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:07:22.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:07:22.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:07:22.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:07:22.056 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:07:22.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:07:22.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:07:22.058 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:07:22.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:07:22.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:07:22.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:07:22.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:07:22.061 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:07:22.061 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:07:22.061 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:07:22.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:07:22.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:07:22.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:07:22.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T22:07:22.064 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:07:22.064 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:07:22.064 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:07:22.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T22:07:22.065 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:07:22.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:07:22.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:07:22.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T22:07:22.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T22:07:22.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T22:07:22.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:07:22.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:07:22.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:07:22.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T22:07:22.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T22:07:22.530 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:07:22.530 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:07:22.530 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:07:22.531 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:07:22.531 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:07:22.531 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:07:22.532 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:07:22.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T22:07:22.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:07:22.734 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:07:22.734 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:07:22.734 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:07:22.734 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:07:22.735 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:07:22.735 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:07:22.735 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:07:22.735 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:07:22.735 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:07:22.736 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:07:22.736 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:07:22.736 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:07:22.736 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:07:22.736 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T22:07:22.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:07:22.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:07:22.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:07:22.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:07:22.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:07:22.850 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:07:22.850 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:07:22.850 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:07:22.850 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:07:22.850 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:07:22.851 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:07:22.851 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:07:22.851 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:07:22.851 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T22:07:22.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T22:07:22.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T22:07:22.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:07:22.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:07:22.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:07:22.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T22:07:22.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:07:23.871 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:07:23.872 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:07:23.872 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:07:23.872 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:07:23.872 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:07:23.872 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:07:23.873 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:07:23.873 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:07:23.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:07:23.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T22:07:23.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T22:07:24.227 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T22:07:24.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:07:25.247 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T22:07:25.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:07:25.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:07:25.248 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:07:25.248 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:07:25.248 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:07:25.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:07:25.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T22:07:25.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T22:07:25.355 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:07:25.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T22:07:25.371 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:07:22.573+0000 7fe618c92e00 0 load: jerasure load: lrc 2024-09-15T22:07:25.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2024-09-15T22:07:25.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2024-09-15T22:07:25.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:501: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2024-09-15T22:07:25.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:07:25.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:07:25.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:505: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2024-09-15T22:07:25.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-15T22:07:25.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T22:07:25.373 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-15T22:07:25.375 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-15T22:07:25.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:07:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T22:07:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:07:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:07:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:07:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:07:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T22:07:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:07:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:07:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:07:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:07:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T22:07:25.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T22:07:25.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:07:25.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:07:25.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:07:25.379 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:07:25.379 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:07:25.379 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:07:25.381 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:07:25.381 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:07:25.381 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:07:25.381 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:07:25.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:07:25.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b52e72d8-3a72-4a52-b20b-392026a30e6f 2024-09-15T22:07:25.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b52e72d8-3a72-4a52-b20b-392026a30e6f' 2024-09-15T22:07:25.383 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 b52e72d8-3a72-4a52-b20b-392026a30e6f 2024-09-15T22:07:25.383 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:07:25.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCdWudmCZQOGBAAD8j/d0SY0LmDJxohAwUeKQ== 2024-09-15T22:07:25.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCdWudmCZQOGBAAD8j/d0SY0LmDJxohAwUeKQ=="}' 2024-09-15T22:07:25.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b52e72d8-3a72-4a52-b20b-392026a30e6f -i td/test-erasure-eio/0/new.json 2024-09-15T22:07:25.695 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:07:25.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T22:07:25.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdWudmCZQOGBAAD8j/d0SY0LmDJxohAwUeKQ== --osd-uuid b52e72d8-3a72-4a52-b20b-392026a30e6f 2024-09-15T22:07:25.758 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:07:25.757+0000 7fd1ecd6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:07:25.763 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:07:25.761+0000 7fd1ecd6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:07:25.765 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:07:25.761+0000 7fd1ecd6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:07:25.765 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:07:25.761+0000 7fd1ecd6a7c0 -1 bdev(0x55ac70de3c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:07:25.765 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:07:25.761+0000 7fd1ecd6a7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T22:07:32.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T22:07:32.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:07:32.066 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T22:07:32.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:07:32.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:07:32.662 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T22:07:32.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:07:32.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:07:32.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:07:32.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:07:32.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:07:32.698 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:07:32.693+0000 7f691c3ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:07:32.710 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:07:32.709+0000 7f691c3ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:07:32.711 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:07:32.709+0000 7f691c3ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:07:33.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:07:33.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:07:33.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:07:33.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:07:33.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:07:33.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:33.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:07:33.099 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:07:33.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:33.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:33.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:33.745 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:07:33.741+0000 7f691c3ff7c0 -1 Falling back to public interface 2024-09-15T22:07:34.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:34.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:34.578 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:07:34.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:07:34.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:34.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:35.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:36.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:36.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:36.059 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:07:36.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:07:36.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:36.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:36.391 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:07:36.389+0000 7f691c3ff7c0 -1 osd.0 0 log_to_monitors true 2024-09-15T22:07:36.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:37.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:37.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:37.489 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:07:37.490 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:07:37.490 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:37.490 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:37.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:38.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:38.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:38.951 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:07:38.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:07:38.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:38.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:39.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:40.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:40.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:40.532 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:07:40.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:07:40.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:40.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:40.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:41.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:41.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:41.983 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:07:41.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:07:41.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:41.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:42.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:43.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:43.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:43.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:07:43.563 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:07:43.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:43.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:44.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:45.025 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:45.025 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:45.025 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:07:45.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:07:45.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:45.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:45.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:46.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:46.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:46.560 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:07:46.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:07:46.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:46.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:47.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:48.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:48.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:48.018 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:07:48.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:07:48.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:48.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:48.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:49.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:49.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:49.515 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:07:49.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:07:49.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:49.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:50.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:51.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:51.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:51.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:07:51.396 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:07:51.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:51.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:52.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:53.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:53.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:53.184 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:07:53.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:07:53.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:53.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:54.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:55.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:55.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:55.012 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:07:55.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:07:55.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:55.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:56.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:57.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:57.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:57.110 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:07:57.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:07:57.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:57.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:57.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:07:58.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:07:58.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:07:58.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:07:58.622 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:07:58.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:07:58.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:07:59.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:00.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:00.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:00.327 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:08:00.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:08:00.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:00.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:08:00.907 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1125095805,v1:127.0.0.1:6803/1125095805] [v2:127.0.0.1:6804/1125095805,v1:127.0.0.1:6805/1125095805] exists,up b52e72d8-3a72-4a52-b20b-392026a30e6f 2024-09-15T22:08:00.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:08:00.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:08:00.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:08:00.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:08:00.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T22:08:00.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:08:00.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:08:00.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:08:00.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:08:00.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:08:00.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:08:00.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:08:00.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:08:00.912 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:08:00.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:08:00.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:08:00.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:08:00.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:08:00.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:08:00.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:08:00.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:08:00.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:08:00.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:08:00.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:08:00.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:08:00.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:08:00.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:08:00.914 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:08:00.914 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 06c0f013-27c8-4155-9bbe-431269719bfb 2024-09-15T22:08:00.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=06c0f013-27c8-4155-9bbe-431269719bfb 2024-09-15T22:08:00.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 06c0f013-27c8-4155-9bbe-431269719bfb' 2024-09-15T22:08:00.915 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:08:00.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDAWudmzhPANxAA8IRJie63DCX3fKRdtOCI/Q== 2024-09-15T22:08:00.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDAWudmzhPANxAA8IRJie63DCX3fKRdtOCI/Q=="}' 2024-09-15T22:08:00.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 06c0f013-27c8-4155-9bbe-431269719bfb -i td/test-erasure-eio/1/new.json 2024-09-15T22:08:01.419 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:08:01.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T22:08:01.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDAWudmzhPANxAA8IRJie63DCX3fKRdtOCI/Q== --osd-uuid 06c0f013-27c8-4155-9bbe-431269719bfb 2024-09-15T22:08:01.470 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:01.469+0000 7fb6c6aa47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:01.472 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:01.469+0000 7fb6c6aa47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:01.474 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:01.469+0000 7fb6c6aa47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:01.474 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:01.473+0000 7fb6c6aa47c0 -1 bdev(0x557c5a42fc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:08:01.474 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:01.473+0000 7fb6c6aa47c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T22:08:07.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T22:08:07.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:08:07.725 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T22:08:07.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:08:07.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:08:08.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:08:08.302 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T22:08:08.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:08:08.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:08:08.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:08:08.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:08:08.338 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:08.337+0000 7fa9af3027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:08.351 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:08.349+0000 7fa9af3027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:08.352 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:08.349+0000 7fa9af3027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:08.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:08:08.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:08:08.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:08:08.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:08:08.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:08:08.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:08.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:08:08.786 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:08:08.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:08.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:09.232 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:09.365 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:09.361+0000 7fa9af3027c0 -1 Falling back to public interface 2024-09-15T22:08:10.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:10.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:10.234 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:08:10.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:08:10.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:10.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:10.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:11.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:11.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:11.702 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:08:11.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:08:11.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:11.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:12.006 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:12.001+0000 7fa9af3027c0 -1 osd.1 0 log_to_monitors true 2024-09-15T22:08:12.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:13.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:13.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:13.225 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:08:13.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:08:13.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:13.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:13.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:14.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:14.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:14.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:08:14.867 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:08:14.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:14.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:15.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:16.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:16.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:16.437 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:08:16.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:08:16.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:16.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:16.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:17.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:17.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:17.961 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:08:17.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:08:17.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:17.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:18.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:19.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:19.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:19.427 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:08:19.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:08:19.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:19.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:19.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:20.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:20.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:20.950 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:08:20.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:08:20.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:20.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:21.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:22.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:22.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:22.447 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:08:22.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:08:22.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:22.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:22.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:23.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:23.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:23.942 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:08:23.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:08:23.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:23.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:24.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:25.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:25.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:25.387 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:08:25.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:08:25.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:25.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:26.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:27.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:27.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:27.174 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:08:27.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:08:27.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:27.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:28.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:29.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:29.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:29.289 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:08:29.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:08:29.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:29.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:30.062 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:31.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:31.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:31.064 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:08:31.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:08:31.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:31.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:31.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:32.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:32.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:32.905 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:08:32.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:08:32.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:32.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:34.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:35.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:35.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:35.235 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:08:35.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:08:35.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:35.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:36.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:36.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:36.808 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:08:36.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:08:36.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:36.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:08:37.286 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/399164218,v1:127.0.0.1:6811/399164218] [v2:127.0.0.1:6812/399164218,v1:127.0.0.1:6813/399164218] exists,up 06c0f013-27c8-4155-9bbe-431269719bfb 2024-09-15T22:08:37.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:08:37.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:08:37.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:08:37.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:08:37.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T22:08:37.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:08:37.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:08:37.288 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:08:37.289 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:08:37.289 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:08:37.292 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:08:37.292 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:08:37.293 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:08:37.293 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:08:37.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:08:37.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:08:37.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:08:37.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:08:37.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:08:37.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:08:37.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:08:37.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:08:37.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:08:37.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:08:37.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:08:37.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:08:37.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:08:37.295 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:08:37.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5fcc1fc2-10a9-42a4-85e6-34be48f8264c 2024-09-15T22:08:37.295 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 5fcc1fc2-10a9-42a4-85e6-34be48f8264c 2024-09-15T22:08:37.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5fcc1fc2-10a9-42a4-85e6-34be48f8264c' 2024-09-15T22:08:37.296 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:08:37.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDlWudmjIHpEhAAnTh0geYNdLF4lUNFMXEB8w== 2024-09-15T22:08:37.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDlWudmjIHpEhAAnTh0geYNdLF4lUNFMXEB8w=="}' 2024-09-15T22:08:37.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5fcc1fc2-10a9-42a4-85e6-34be48f8264c -i td/test-erasure-eio/2/new.json 2024-09-15T22:08:37.895 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:08:37.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T22:08:37.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlWudmjIHpEhAAnTh0geYNdLF4lUNFMXEB8w== --osd-uuid 5fcc1fc2-10a9-42a4-85e6-34be48f8264c 2024-09-15T22:08:37.948 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:37.944+0000 7fae822137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:37.950 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:37.948+0000 7fae822137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:37.952 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:37.948+0000 7fae822137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:37.952 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:37.948+0000 7fae822137c0 -1 bdev(0x556b75f7bc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:08:37.952 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:37.948+0000 7fae822137c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T22:08:44.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T22:08:44.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:08:44.689 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T22:08:44.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:08:44.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:08:45.282 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:08:45.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:08:45.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:08:45.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:08:45.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:08:45.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:08:45.320 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:45.316+0000 7f39edc557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:45.322 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:45.320+0000 7f39edc557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:45.323 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:45.320+0000 7f39edc557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:08:45.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:08:45.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:08:45.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:08:45.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:08:45.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:08:45.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:45.758 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:08:45.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:08:45.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:45.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:08:46.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:46.837 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:46.832+0000 7f39edc557c0 -1 Falling back to public interface 2024-09-15T22:08:47.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:47.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:47.209 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:08:47.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:08:47.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:47.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:08:47.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:48.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:48.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:48.654 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:08:48.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:08:48.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:48.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:08:49.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:49.667 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:08:49.664+0000 7f39edc557c0 -1 osd.2 0 log_to_monitors true 2024-09-15T22:08:50.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:50.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:50.123 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:08:50.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:08:50.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:50.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:08:50.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:51.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:51.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:51.795 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:08:51.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:08:51.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:51.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:08:52.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:53.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:53.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:53.248 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:08:53.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:08:53.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:53.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:08:53.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:54.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:54.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:54.804 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:08:54.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:08:54.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:54.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:08:55.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:56.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:56.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:56.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:08:56.376 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:08:56.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:56.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:08:56.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:57.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:57.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:57.835 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:08:57.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:08:57.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:57.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:08:58.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:08:59.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:08:59.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:08:59.287 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:08:59.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:08:59.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:08:59.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:08:59.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:00.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:00.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:00.860 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:09:00.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:09:00.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:00.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:09:01.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:02.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:02.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:02.391 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:09:02.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:09:02.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:02.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:09:03.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:04.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:04.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:04.087 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:09:04.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:09:04.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:04.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:09:04.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:05.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:05.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:05.852 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:09:05.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:09:05.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:05.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:09:06.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:07.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:07.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:07.705 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:09:07.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:09:07.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:07.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:09:08.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:09.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:09.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:09.773 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:09:09.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:09:09.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:09.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:09:10.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:11.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:11.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:11.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:09:11.811 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:09:11.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:11.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:09:12.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:13.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:13.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:13.367 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:09:13.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:09:13.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:13.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:09:13.840 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2556125495,v1:127.0.0.1:6819/2556125495] [v2:127.0.0.1:6820/2556125495,v1:127.0.0.1:6821/2556125495] exists,up 5fcc1fc2-10a9-42a4-85e6-34be48f8264c 2024-09-15T22:09:13.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:09:13.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:09:13.844 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:09:13.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:09:13.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:09:13.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:09:13.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:09:13.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:09:13.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:09:13.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:09:13.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:09:13.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:09:13.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:09:13.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:09:13.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:09:13.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:09:13.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:09:13.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e580152d-6488-41ab-a56c-bbca5d64d79a 2024-09-15T22:09:13.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 e580152d-6488-41ab-a56c-bbca5d64d79a' 2024-09-15T22:09:13.847 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 e580152d-6488-41ab-a56c-bbca5d64d79a 2024-09-15T22:09:13.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:09:13.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAJW+dmOeL9MxAAH60rlFYr6xJ6niOr0mV04g== 2024-09-15T22:09:13.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAJW+dmOeL9MxAAH60rlFYr6xJ6niOr0mV04g=="}' 2024-09-15T22:09:13.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e580152d-6488-41ab-a56c-bbca5d64d79a -i td/test-erasure-eio/3/new.json 2024-09-15T22:09:14.749 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:09:14.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T22:09:14.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJW+dmOeL9MxAAH60rlFYr6xJ6niOr0mV04g== --osd-uuid e580152d-6488-41ab-a56c-bbca5d64d79a 2024-09-15T22:09:14.806 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:14.800+0000 7f3fbc4ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:09:14.860 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:14.856+0000 7f3fbc4ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:09:14.862 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:14.860+0000 7f3fbc4ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:09:14.863 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:14.860+0000 7f3fbc4ab7c0 -1 bdev(0x561fd4ef3c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T22:09:14.863 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:14.860+0000 7f3fbc4ab7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T22:09:21.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T22:09:21.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:09:21.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T22:09:21.845 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T22:09:21.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:09:22.419 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T22:09:22.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T22:09:22.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:09:22.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:09:22.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:09:22.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:09:22.456 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:22.452+0000 7f985160d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:09:22.461 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:22.456+0000 7f985160d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:09:22.462 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:22.460+0000 7f985160d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:09:22.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T22:09:22.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:09:22.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:09:22.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:09:22.854 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:09:22.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:09:22.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:22.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:09:22.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:22.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:23.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:23.685 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:23.680+0000 7f985160d7c0 -1 Falling back to public interface 2024-09-15T22:09:24.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:24.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:24.352 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:09:24.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:09:24.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:24.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:24.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:25.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:25.860 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:09:25.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:25.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:09:25.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:25.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:26.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:26.511 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:26.508+0000 7f985160d7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T22:09:27.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:27.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:27.284 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:09:27.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:09:27.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:27.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:28.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:29.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:29.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:29.053 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:09:29.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:09:29.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:29.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:29.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:30.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:30.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:30.747 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:09:30.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:09:30.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:30.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:31.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:32.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:32.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:32.264 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:09:32.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:09:32.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:32.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:32.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:33.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:33.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:33.793 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:09:33.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:09:33.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:33.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:34.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:35.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:35.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:35.236 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:09:35.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:09:35.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:35.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:35.776 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:36.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:36.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:36.778 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:09:36.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:09:36.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:36.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:37.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:38.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:38.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:38.350 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:09:38.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:09:38.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:38.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:38.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:39.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:39.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:39.800 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:09:39.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:09:39.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:39.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:40.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:41.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:41.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:41.423 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:09:41.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:09:41.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:41.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:42.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:43.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:43.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:43.382 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:09:43.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:09:43.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:43.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:44.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:45.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:45.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:45.372 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:09:45.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:09:45.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:45.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:45.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:46.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:46.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:46.867 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:09:46.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:09:46.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:46.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:47.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:48.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:48.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:48.367 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:09:48.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:09:48.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:48.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:49.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:50.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:50.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:50.376 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:09:50.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:09:50.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:50.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:51.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:09:52.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:09:52.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:09:52.056 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:09:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:09:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:09:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:09:52.615 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3741956856,v1:127.0.0.1:6827/3741956856] [v2:127.0.0.1:6828/3741956856,v1:127.0.0.1:6829/3741956856] exists,up e580152d-6488-41ab-a56c-bbca5d64d79a 2024-09-15T22:09:52.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:09:52.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:09:52.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T22:09:52.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T22:09:52.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:09:52.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:09:52.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:09:52.619 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:09:52.619 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:09:52.619 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:09:52.619 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:09:52.619 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:09:52.619 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:09:52.621 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:09:52.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:09:52.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:09:52.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:09:52.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:09:52.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:09:52.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:09:52.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:09:52.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:09:52.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:09:52.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:09:52.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:09:52.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:09:52.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T22:09:52.622 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:09:52.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c28c6921-aa15-4715-a3c0-cb12911acd68 2024-09-15T22:09:52.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 c28c6921-aa15-4715-a3c0-cb12911acd68' 2024-09-15T22:09:52.623 INFO:tasks.workunit.client.0.smithi104.stdout:add osd4 c28c6921-aa15-4715-a3c0-cb12911acd68 2024-09-15T22:09:52.623 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:09:52.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAwW+dmcOC9JhAA+qZeekqLqf6YgZ+LjGM9FA== 2024-09-15T22:09:52.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAwW+dmcOC9JhAA+qZeekqLqf6YgZ+LjGM9FA=="}' 2024-09-15T22:09:52.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c28c6921-aa15-4715-a3c0-cb12911acd68 -i td/test-erasure-eio/4/new.json 2024-09-15T22:09:53.611 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:09:53.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T22:09:53.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAwW+dmcOC9JhAA+qZeekqLqf6YgZ+LjGM9FA== --osd-uuid c28c6921-aa15-4715-a3c0-cb12911acd68 2024-09-15T22:09:53.662 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:53.660+0000 7ff3661537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:09:53.695 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:53.692+0000 7ff3661537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:09:53.697 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:53.692+0000 7ff3661537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:09:53.697 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:53.692+0000 7ff3661537c0 -1 bdev(0x561f77089c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-15T22:09:53.697 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:09:53.692+0000 7ff3661537c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T22:09:59.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T22:09:59.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:09:59.957 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd4 key to auth repository 2024-09-15T22:09:59.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-15T22:09:59.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:10:00.557 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.4 2024-09-15T22:10:00.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-15T22:10:00.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:10:00.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:10:00.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:10:00.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:10:00.594 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:00.596+0000 7fdeecca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:10:00.596 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:00.596+0000 7fdeecca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:10:00.597 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:00.596+0000 7fdeecca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:10:00.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-15T22:10:00.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:10:00.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-15T22:10:00.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:10:00.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:10:00.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:00.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:10:00.996 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:10:00.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:00.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:01.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:01.857 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:01.856+0000 7fdeecca47c0 -1 Falling back to public interface 2024-09-15T22:10:02.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:02.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:02.466 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:10:02.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:10:02.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:02.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:02.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:03.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:03.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:03.950 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:10:03.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:10:03.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:03.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:04.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:04.698 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:04.700+0000 7fdeecca47c0 -1 osd.4 0 log_to_monitors true 2024-09-15T22:10:05.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:05.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:05.398 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:10:05.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:10:05.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:05.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:06.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:07.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:07.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:07.148 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:10:07.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:10:07.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:07.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:08.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:09.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:09.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:09.041 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:10:09.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:10:09.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:09.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:09.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:10.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:10.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:10.735 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:10:10.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:10:10.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:10.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:11.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:12.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:12.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:12.274 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:10:12.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:10:12.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:12.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:13.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:13.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:13.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:13.999 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:10:14.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:10:14.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:14.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:14.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:15.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:15.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:15.525 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:10:15.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:10:15.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:15.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:15.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:16.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:16.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:16.973 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:10:16.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:10:16.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:16.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:17.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:18.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:18.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:18.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:10:18.730 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:10:18.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:18.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:19.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:20.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:20.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:20.402 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:10:20.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:10:20.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:20.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:21.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:22.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:22.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:22.632 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:10:22.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:10:22.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:22.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:24.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:25.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:25.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:25.020 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:10:25.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:10:25.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:25.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:25.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:26.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:26.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:26.960 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:10:26.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:10:26.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:26.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:27.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:28.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:28.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:28.735 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:10:28.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:10:28.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:28.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:29.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:30.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:30.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:30.900 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:10:30.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:10:30.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:30.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:10:31.541 INFO:tasks.workunit.client.0.smithi104.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2922341635,v1:127.0.0.1:6835/2922341635] [v2:127.0.0.1:6836/2922341635,v1:127.0.0.1:6837/2922341635] exists,up c28c6921-aa15-4715-a3c0-cb12911acd68 2024-09-15T22:10:31.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:10:31.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:10:31.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:10:31.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:10:31.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-15T22:10:31.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:10:31.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:10:31.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-15T22:10:31.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:10:31.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-15T22:10:31.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:10:31.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:10:31.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:10:31.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:10:31.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-15T22:10:31.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-15T22:10:31.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:10:31.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:10:31.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:10:31.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:10:31.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:10:31.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:10:31.545 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:10:31.545 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:10:31.545 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:10:31.546 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:10:31.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-15T22:10:31.547 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:10:31.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ab4889c4-9afd-4cee-a4dc-b13db6ac7f31 2024-09-15T22:10:31.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 ab4889c4-9afd-4cee-a4dc-b13db6ac7f31' 2024-09-15T22:10:31.548 INFO:tasks.workunit.client.0.smithi104.stdout:add osd5 ab4889c4-9afd-4cee-a4dc-b13db6ac7f31 2024-09-15T22:10:31.548 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:10:31.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBXW+dmdsksIhAAbpc97ZACqQSS/I7rB/Mkmg== 2024-09-15T22:10:31.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBXW+dmdsksIhAAbpc97ZACqQSS/I7rB/Mkmg=="}' 2024-09-15T22:10:31.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ab4889c4-9afd-4cee-a4dc-b13db6ac7f31 -i td/test-erasure-eio/5/new.json 2024-09-15T22:10:32.230 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:10:32.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-15T22:10:32.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXW+dmdsksIhAAbpc97ZACqQSS/I7rB/Mkmg== --osd-uuid ab4889c4-9afd-4cee-a4dc-b13db6ac7f31 2024-09-15T22:10:32.278 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:32.275+0000 7f9a72e347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:10:32.280 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:32.279+0000 7f9a72e347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:10:32.281 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:32.279+0000 7f9a72e347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:10:32.281 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:32.279+0000 7f9a72e347c0 -1 bdev(0x5598c9093c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-15T22:10:32.281 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:32.279+0000 7f9a72e347c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-15T22:10:38.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-15T22:10:38.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:10:38.664 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd5 key to auth repository 2024-09-15T22:10:38.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-15T22:10:38.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:10:39.298 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.5 2024-09-15T22:10:39.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-15T22:10:39.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:10:39.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:10:39.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:10:39.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:10:39.335 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:39.335+0000 7fdfe43d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:10:39.337 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:39.335+0000 7fdfe43d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:10:39.339 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:39.339+0000 7fdfe43d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:10:39.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-15T22:10:39.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:10:39.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T22:10:39.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:10:39.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:10:39.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:39.715 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:10:39.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:10:39.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:39.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:40.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:40.817 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:40.815+0000 7fdfe43d17c0 -1 Falling back to public interface 2024-09-15T22:10:41.131 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:41.131 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:41.131 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:10:41.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:10:41.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:41.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:41.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:42.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:42.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:42.577 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:10:42.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:10:42.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:42.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:43.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:43.746 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:10:43.747+0000 7fdfe43d17c0 -1 osd.5 0 log_to_monitors true 2024-09-15T22:10:44.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:44.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:44.093 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:10:44.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:10:44.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:44.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:44.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:45.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:45.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:45.667 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:10:45.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:10:45.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:45.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:46.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:47.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:47.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:47.576 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:10:47.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:10:47.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:47.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:48.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:49.131 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:49.131 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:49.131 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:10:49.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:10:49.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:49.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:49.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:50.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:50.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:50.731 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:10:50.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:10:50.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:50.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:51.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:52.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:52.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:52.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:10:52.278 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:10:52.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:52.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:52.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:53.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:53.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:53.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:10:53.723 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:10:53.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:53.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:54.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:55.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:55.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:55.168 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:10:55.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:10:55.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:55.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:55.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:56.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:56.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:56.637 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:10:56.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:10:56.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:56.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:57.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:58.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:58.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:58.073 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:10:58.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:10:58.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:58.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:10:58.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:10:59.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:10:59.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:10:59.654 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:10:59.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:10:59.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:10:59.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:11:00.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:01.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:01.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:01.755 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:11:01.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:11:01.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:01.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:11:02.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:03.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:03.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:03.507 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:11:03.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:11:03.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:03.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:11:04.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:05.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:05.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:05.300 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:11:05.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:11:05.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:05.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:11:06.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:07.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:07.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:07.328 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:11:07.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:11:07.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:07.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:11:07.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:08.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:08.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:08.965 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:11:08.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:11:08.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:08.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:11:09.481 INFO:tasks.workunit.client.0.smithi104.stdout:osd.5 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2656680363,v1:127.0.0.1:6843/2656680363] [v2:127.0.0.1:6844/2656680363,v1:127.0.0.1:6845/2656680363] exists,up ab4889c4-9afd-4cee-a4dc-b13db6ac7f31 2024-09-15T22:11:09.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:11:09.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:11:09.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:11:09.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:11:09.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-15T22:11:09.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:11:09.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:11:09.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-15T22:11:09.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:11:09.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:11:09.486 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:11:09.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:11:09.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:11:09.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-15T22:11:09.488 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:11:09.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d010dd55-5556-4a4b-bbb0-f7da4ef5ef9d 2024-09-15T22:11:09.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 d010dd55-5556-4a4b-bbb0-f7da4ef5ef9d' 2024-09-15T22:11:09.489 INFO:tasks.workunit.client.0.smithi104.stdout:add osd6 d010dd55-5556-4a4b-bbb0-f7da4ef5ef9d 2024-09-15T22:11:09.489 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:11:09.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB9W+dm4JqkHhAAqgQoeoFpQqhI6WrI+nU/QA== 2024-09-15T22:11:09.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB9W+dm4JqkHhAAqgQoeoFpQqhI6WrI+nU/QA=="}' 2024-09-15T22:11:09.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d010dd55-5556-4a4b-bbb0-f7da4ef5ef9d -i td/test-erasure-eio/6/new.json 2024-09-15T22:11:10.012 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:11:10.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-15T22:11:10.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9W+dm4JqkHhAAqgQoeoFpQqhI6WrI+nU/QA== --osd-uuid d010dd55-5556-4a4b-bbb0-f7da4ef5ef9d 2024-09-15T22:11:10.064 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:11:10.063+0000 7ff8dce357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:11:10.179 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:11:10.179+0000 7ff8dce357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:11:10.180 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:11:10.179+0000 7ff8dce357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:11:10.181 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:11:10.179+0000 7ff8dce357c0 -1 bdev(0x558b87b53c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-15T22:11:10.181 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:11:10.179+0000 7ff8dce357c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-15T22:11:16.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-15T22:11:16.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:11:16.889 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd6 key to auth repository 2024-09-15T22:11:16.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-15T22:11:16.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:11:17.490 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-15T22:11:17.490 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.6 2024-09-15T22:11:17.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:11:17.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:11:17.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:11:17.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:11:17.526 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:11:17.527+0000 7fdc924e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:11:17.528 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:11:17.527+0000 7fdc924e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:11:17.529 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:11:17.527+0000 7fdc924e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:11:17.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-15T22:11:17.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:11:17.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-15T22:11:17.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:11:17.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:11:17.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:17.926 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:11:17.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:11:17.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:17.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:18.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:18.745 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:11:18.743+0000 7fdc924e67c0 -1 Falling back to public interface 2024-09-15T22:11:19.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:19.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:19.402 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:11:19.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:11:19.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:19.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:19.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:20.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:20.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:20.895 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:11:20.895 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:11:20.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:20.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:21.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:21.642 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:11:21.643+0000 7fdc924e67c0 -1 osd.6 0 log_to_monitors true 2024-09-15T22:11:22.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:22.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:22.395 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:11:22.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:11:22.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:22.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:22.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:23.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:23.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:23.868 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:11:23.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:11:23.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:23.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:24.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:25.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:25.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:25.531 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:11:25.531 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:11:25.531 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:25.531 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:26.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:27.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:27.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:27.160 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:11:27.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:11:27.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:27.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:27.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:28.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:28.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:28.834 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:11:28.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:11:28.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:28.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:29.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:30.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:30.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:30.443 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:11:30.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:11:30.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:30.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:30.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:31.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:31.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:31.903 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:11:31.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:11:31.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:31.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:32.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:33.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:33.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:33.495 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:11:33.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:11:33.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:33.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:33.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:34.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:34.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:34.951 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:11:34.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:11:34.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:34.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:35.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:36.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:36.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:36.425 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:11:36.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:11:36.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:36.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:37.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:38.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:38.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:38.348 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:11:38.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:11:38.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:38.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:39.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:40.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:40.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:40.356 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:11:40.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:11:40.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:40.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:42.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:42.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:42.354 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:11:42.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:11:42.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:42.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:43.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:44.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:44.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:44.296 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:11:44.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:11:44.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:44.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:45.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:46.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:46.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:46.076 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:11:46.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:11:46.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:46.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:46.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:47.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:47.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:47.763 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:11:47.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:11:47.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:47.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:48.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:11:49.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:11:49.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:11:49.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T22:11:49.335 INFO:tasks.workunit.client.0.smithi104.stdout:19 2024-09-15T22:11:49.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:11:49.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:11:49.861 INFO:tasks.workunit.client.0.smithi104.stdout:osd.6 up in weight 1 up_from 59 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1731659845,v1:127.0.0.1:6851/1731659845] [v2:127.0.0.1:6852/1731659845,v1:127.0.0.1:6853/1731659845] exists,up d010dd55-5556-4a4b-bbb0-f7da4ef5ef9d 2024-09-15T22:11:49.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:11:49.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:11:49.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:11:49.863 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:11:49.863 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:11:49.863 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:11:49.863 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:11:49.863 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:11:49.863 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:11:49.863 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:11:49.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:11:49.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:11:49.977 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:11:49.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:11:49.998 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:07:33.753+0000 7f691c3ff7c0 0 load: jerasure load: lrc 2024-09-15T22:11:49.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:11:49.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:508: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2024-09-15T22:11:49.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:509: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T22:11:49.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:11:49.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:11:50.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T22:11:50.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:11:50.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T22:11:50.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:11:50.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T22:11:50.571 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:11:50.571 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:11:51.703 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:11:51.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:11:52.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:11:52.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:11:52.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:11:52.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:11:52.724 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:11:52.725 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:11:52.725 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:11:52.726 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:11:52.726 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:11:52.726 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:11:52.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:11:52.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:11:52.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:11:52.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:11:52.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:11:52.852 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:11:53.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:11:53.424 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:11:53.424 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:11:53.424 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:11:53.424 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:11:53.424 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:11:53.425 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T22:11:53.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:11:53.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:11:53.426 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:11:53.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803826 2024-09-15T22:11:53.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803826 2024-09-15T22:11:53.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826' 2024-09-15T22:11:53.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:11:53.582 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:11:53.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574895 2024-09-15T22:11:53.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574895 2024-09-15T22:11:53.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826 1-55834574895' 2024-09-15T22:11:53.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:11:53.735 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:11:53.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345956 2024-09-15T22:11:53.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345956 2024-09-15T22:11:53.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826 1-55834574895 2-85899345956' 2024-09-15T22:11:53.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:11:53.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:11:54.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018909 2024-09-15T22:11:54.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018909 2024-09-15T22:11:54.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826 1-55834574895 2-85899345956 3-128849018909' 2024-09-15T22:11:54.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:11:54.031 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T22:11:54.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659161 2024-09-15T22:11:54.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659161 2024-09-15T22:11:54.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826 1-55834574895 2-85899345956 3-128849018909 4-176093659161' 2024-09-15T22:11:54.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:11:54.174 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T22:11:54.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364820 2024-09-15T22:11:54.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364820 2024-09-15T22:11:54.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826 1-55834574895 2-85899345956 3-128849018909 4-176093659161 5-214748364820' 2024-09-15T22:11:54.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:11:54.324 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T22:11:54.479 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=253403070474 2024-09-15T22:11:54.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 253403070474 2024-09-15T22:11:54.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826 1-55834574895 2-85899345956 3-128849018909 4-176093659161 5-214748364820 6-253403070474' 2024-09-15T22:11:54.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:11:54.480 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803826 2024-09-15T22:11:54.480 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:11:54.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:11:54.482 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803826 2024-09-15T22:11:54.482 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:11:54.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803826 2024-09-15T22:11:54.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-09-15T22:11:54.483 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803826 2024-09-15T22:11:54.484 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:11:54.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803826 -lt 25769803826 2024-09-15T22:11:54.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:11:54.905 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:11:54.905 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574895 2024-09-15T22:11:54.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:11:54.907 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574895 2024-09-15T22:11:54.907 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:11:54.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574895 2024-09-15T22:11:54.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574895' 2024-09-15T22:11:54.909 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574895 2024-09-15T22:11:54.909 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:11:55.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574895 -lt 55834574895 2024-09-15T22:11:55.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:11:55.333 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345956 2024-09-15T22:11:55.333 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:11:55.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:11:55.335 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345956 2024-09-15T22:11:55.335 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:11:55.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345956 2024-09-15T22:11:55.337 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 85899345956 2024-09-15T22:11:55.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345956' 2024-09-15T22:11:55.337 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:11:55.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345957 -lt 85899345956 2024-09-15T22:11:55.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:11:55.767 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:11:55.767 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018909 2024-09-15T22:11:55.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:11:55.769 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018909 2024-09-15T22:11:55.769 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:11:55.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018909 2024-09-15T22:11:55.771 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 128849018909 2024-09-15T22:11:55.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018909' 2024-09-15T22:11:55.771 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:11:56.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018909 -lt 128849018909 2024-09-15T22:11:56.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:11:56.182 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-176093659161 2024-09-15T22:11:56.182 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:11:56.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T22:11:56.184 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-176093659161 2024-09-15T22:11:56.184 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:11:56.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659161 2024-09-15T22:11:56.185 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 176093659161 2024-09-15T22:11:56.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 176093659161' 2024-09-15T22:11:56.186 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T22:11:56.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659161 -lt 176093659161 2024-09-15T22:11:56.652 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:11:56.653 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-214748364820 2024-09-15T22:11:56.653 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:11:56.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T22:11:56.655 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-214748364820 2024-09-15T22:11:56.655 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:11:56.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364820 2024-09-15T22:11:56.657 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 214748364820 2024-09-15T22:11:56.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 214748364820' 2024-09-15T22:11:56.657 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T22:11:57.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364820 -lt 214748364820 2024-09-15T22:11:57.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:11:57.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-253403070474 2024-09-15T22:11:57.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:11:57.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T22:11:57.118 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-253403070474 2024-09-15T22:11:57.118 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:11:57.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=253403070474 2024-09-15T22:11:57.119 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 253403070474 2024-09-15T22:11:57.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 253403070474' 2024-09-15T22:11:57.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T22:11:57.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 253403070474 -lt 253403070474 2024-09-15T22:11:57.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:11:57.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:11:57.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:11:58.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:11:58.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:11:58.168 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:11:58.169 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:11:58.169 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:11:58.169 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:11:58.169 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:11:58.169 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:11:58.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:11:58.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:11:58.582 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:11:58.582 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:11:59.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:11:59.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:11:59.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-15T22:11:59.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-15T22:11:59.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:11:59.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:11:59.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-15T22:11:59.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:11:59.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:11:59.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:11:59.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:11:59.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-15T22:11:59.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-15T22:11:59.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:11:59.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:11:59.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-15T22:11:59.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:11:59.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:11:59.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:11:59.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:11:59.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:11:59.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-15T22:11:59.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:516: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-15T22:11:59.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:11:59.959 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-15T22:11:59.960 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:11:59.960 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-15T22:11:59.960 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T22:11:59.960 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:12:00.416 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:12:00.416 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:12:00.416 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:12:00.416 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:12:00.416 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T22:12:00.416 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2024-09-15T22:12:00.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-09-15T22:12:00.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:12:00.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-15T22:12:00.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:12:00.417 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T22:12:00.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:12:00.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T22:12:00.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:12:00.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-15T22:12:00.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:12:00.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:12:00.419 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:12:00.419 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-15T22:12:00.419 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:12:00.419 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:12:00.420 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:12:00.420 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:00.420 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:00.420 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:12:00.420 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:12:00.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:12:00.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:12:00.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:12:00.541 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-15T22:12:00.541 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:12:00.542 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:12:00.542 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:12:00.542 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:00.542 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:00.542 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:12:00.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:12:00.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-15T22:12:00.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2024-09-15T22:12:01.308 INFO:tasks.workunit.client.0.smithi104.stderr:osd.0 is already out. 2024-09-15T22:12:01.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2024-09-15T22:12:01.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:12:01.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:12:01.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:12:01.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:12:01.328 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:12:01.328 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:12:01.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:12:01.329 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:12:01.329 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:12:01.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:12:01.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:12:01.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:12:01.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:12:01.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:12:01.443 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:12:02.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:12:02.145 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:12:02.145 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:12:02.146 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:12:02.146 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:12:02.146 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:12:02.146 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T22:12:02.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:12:02.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:12:02.146 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:12:02.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803830 2024-09-15T22:12:02.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803830 2024-09-15T22:12:02.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803830' 2024-09-15T22:12:02.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:12:02.372 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:12:02.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574899 2024-09-15T22:12:02.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574899 2024-09-15T22:12:02.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803830 1-55834574899' 2024-09-15T22:12:02.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:12:02.660 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:12:02.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345960 2024-09-15T22:12:02.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345960 2024-09-15T22:12:02.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803830 1-55834574899 2-85899345960' 2024-09-15T22:12:02.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:12:02.809 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:12:03.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018913 2024-09-15T22:12:03.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018913 2024-09-15T22:12:03.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803830 1-55834574899 2-85899345960 3-128849018913' 2024-09-15T22:12:03.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:12:03.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T22:12:03.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659165 2024-09-15T22:12:03.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659165 2024-09-15T22:12:03.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803830 1-55834574899 2-85899345960 3-128849018913 4-176093659165' 2024-09-15T22:12:03.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:12:03.278 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T22:12:03.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364823 2024-09-15T22:12:03.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364823 2024-09-15T22:12:03.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803830 1-55834574899 2-85899345960 3-128849018913 4-176093659165 5-214748364823' 2024-09-15T22:12:03.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:12:03.559 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T22:12:03.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=253403070477 2024-09-15T22:12:03.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 253403070477 2024-09-15T22:12:03.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803830 1-55834574899 2-85899345960 3-128849018913 4-176093659165 5-214748364823 6-253403070477' 2024-09-15T22:12:03.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:12:03.717 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803830 2024-09-15T22:12:03.717 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:12:03.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:12:03.719 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803830 2024-09-15T22:12:03.719 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:12:03.720 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803830 2024-09-15T22:12:03.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803830 2024-09-15T22:12:03.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-09-15T22:12:03.721 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:12:04.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803830 -lt 25769803830 2024-09-15T22:12:04.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:12:04.213 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574899 2024-09-15T22:12:04.213 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:12:04.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:12:04.215 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574899 2024-09-15T22:12:04.216 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:12:04.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574899 2024-09-15T22:12:04.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574899' 2024-09-15T22:12:04.218 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574899 2024-09-15T22:12:04.218 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:12:04.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574899 -lt 55834574899 2024-09-15T22:12:04.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:12:04.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:12:04.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345960 2024-09-15T22:12:04.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:12:04.636 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345960 2024-09-15T22:12:04.636 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:12:04.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345960 2024-09-15T22:12:04.638 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 85899345960 2024-09-15T22:12:04.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345960' 2024-09-15T22:12:04.639 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:12:05.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345961 -lt 85899345960 2024-09-15T22:12:05.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:12:05.140 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018913 2024-09-15T22:12:05.140 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:12:05.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:12:05.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018913 2024-09-15T22:12:05.142 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:12:05.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018913 2024-09-15T22:12:05.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018913' 2024-09-15T22:12:05.143 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 128849018913 2024-09-15T22:12:05.143 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:12:05.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018913 -lt 128849018913 2024-09-15T22:12:05.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:12:05.577 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-176093659165 2024-09-15T22:12:05.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:12:05.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T22:12:05.580 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-176093659165 2024-09-15T22:12:05.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:12:05.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659165 2024-09-15T22:12:05.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 176093659165' 2024-09-15T22:12:05.581 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 176093659165 2024-09-15T22:12:05.582 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T22:12:06.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659165 -lt 176093659165 2024-09-15T22:12:06.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:12:06.062 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-214748364823 2024-09-15T22:12:06.062 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:12:06.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T22:12:06.064 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-214748364823 2024-09-15T22:12:06.064 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:12:06.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364823 2024-09-15T22:12:06.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 214748364823' 2024-09-15T22:12:06.066 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 214748364823 2024-09-15T22:12:06.066 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T22:12:06.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364824 -lt 214748364823 2024-09-15T22:12:06.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:12:06.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-253403070477 2024-09-15T22:12:06.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:12:06.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T22:12:06.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-253403070477 2024-09-15T22:12:06.495 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:12:06.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=253403070477 2024-09-15T22:12:06.496 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 253403070477 2024-09-15T22:12:06.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 253403070477' 2024-09-15T22:12:06.496 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T22:12:06.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 253403070478 -lt 253403070477 2024-09-15T22:12:06.989 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:12:06.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:12:06.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:12:07.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:12:07.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:12:07.570 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:12:07.570 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:12:07.570 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:12:07.570 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:12:07.570 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:12:07.570 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:12:07.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T22:12:07.992 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:12:07.992 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:12:07.992 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:12:08.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T22:12:08.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T22:12:08.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T22:12:08.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T22:12:08.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T22:12:08.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T22:12:08.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T22:12:08.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:12:08.647 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:12:08.647 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:12:08.647 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:12:08.648 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:12:08.648 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:12:08.648 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:12:09.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:12:09.072 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:12:09.073 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:12:09.073 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:12:09.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:12:09.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:12:09.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:12:09.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-15T22:12:09.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:12:09.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:12:09.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-15T22:12:09.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:12:09.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:12:09.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-15T22:12:09.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:12:09.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:12:09.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-15T22:12:09.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:12:09.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:12:09.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-15T22:12:09.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:12:09.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:12:09.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-15T22:12:09.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:12:09.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:12:09.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:524: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-15T22:12:09.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:12:09.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:12:09.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-15T22:12:09.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:12:09.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:12:09.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-15T22:12:09.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:12:09.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:12:09.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:526: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:12:09.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:12:09.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:12:10.272 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:12:10.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:12:10.908 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:12:10.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:12:10.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:12:10.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:12:10.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:12:10.926 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:12:10.926 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:12:10.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:12:10.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:12:10.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:12:11.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:12:11.239 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:12:11.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:12:11.241 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:12:11.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:12:11.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:12:11.243 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:12:11.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:12:11.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:12:11.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:12:11.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:12:11.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:12:11.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:12:11.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:12:11.288 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:12:11.288 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:11.288 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:11.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:12:11.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:12:11.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:12:11.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T22:12:11.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T22:12:11.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T22:12:11.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T22:12:11.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:12:11.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:12:11.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:12:11.290 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:12:11.290 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:12:11.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:12:11.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:12:11.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:12:11.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:12:11.293 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:12:11.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:12:11.294 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:12:11.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:12:11.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:12:11.295 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:12:11.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:12:11.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:12:11.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:12:11.297 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:12:11.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:12:11.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:12:11.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:12:11.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:12:11.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:11.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:11.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:12:11.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:12:11.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:12:11.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T22:12:11.303 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:12:11.303 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:11.303 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:11.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T22:12:11.305 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:12:11.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:12:11.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:12:11.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T22:12:11.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T22:12:11.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T22:12:11.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:12:11.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:12:11.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:12:11.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T22:12:11.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T22:12:11.767 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:12:11.767 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:12:11.767 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:12:11.768 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:12:11.768 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:11.768 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:11.768 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:12:11.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T22:12:11.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:12:11.994 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:12:11.994 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:12:11.994 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:12:11.994 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:12:11.995 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:12:11.995 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:12:11.995 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:12:11.995 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:12:11.996 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:12:11.996 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:11.996 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:11.996 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:12:11.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:12:11.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T22:12:12.116 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:12:12.117 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:12:12.117 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:12:12.117 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:12:12.118 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:12:12.118 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:12:12.118 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:12:12.118 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:12:12.118 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:12:12.118 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:12.118 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:12.118 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:12:12.119 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:12:12.119 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T22:12:12.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T22:12:12.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T22:12:12.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:12:12.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:12:12.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:12:12.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T22:12:12.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:12:13.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:12:13.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:12:13.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:12:13.136 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:12:13.136 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:13.136 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:13.136 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:12:13.136 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:12:13.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:12:13.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T22:12:13.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T22:12:13.449 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T22:12:13.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:12:14.467 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T22:12:14.467 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:12:14.467 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:12:14.467 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:12:14.467 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:14.467 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:14.467 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:12:14.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T22:12:14.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T22:12:14.583 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:12:14.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T22:12:14.599 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:12:11.810+0000 7fda6fa65e00 0 load: jerasure load: lrc 2024-09-15T22:12:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2024-09-15T22:12:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2024-09-15T22:12:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: local objname=myobject 2024-09-15T22:12:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: local lastobj=100 2024-09-15T22:12:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:633: TEST_ec_recovery_unfound: local testobj=obj75 2024-09-15T22:12:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:635: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:12:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T22:12:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T22:12:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:638: TEST_ec_recovery_unfound: setup_osds 5 2024-09-15T22:12:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-15T22:12:14.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T22:12:14.602 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-15T22:12:14.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-15T22:12:14.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:12:14.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T22:12:14.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:12:14.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:12:14.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:12:14.608 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:12:14.608 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:12:14.608 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:12:14.608 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:12:14.608 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:14.608 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:14.608 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:12:14.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:12:14.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:12:14.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:12:14.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:12:14.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:12:14.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:12:14.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:12:14.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:12:14.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:12:14.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:12:14.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:12:14.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:12:14.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T22:12:14.610 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:12:14.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=54ee31a7-a333-4fa1-b42f-903d8066c21d 2024-09-15T22:12:14.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 54ee31a7-a333-4fa1-b42f-903d8066c21d' 2024-09-15T22:12:14.610 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 54ee31a7-a333-4fa1-b42f-903d8066c21d 2024-09-15T22:12:14.611 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:12:14.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC+W+dm2m6yJRAASwXOFRCdorjcSgxF2k+ZMw== 2024-09-15T22:12:14.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC+W+dm2m6yJRAASwXOFRCdorjcSgxF2k+ZMw=="}' 2024-09-15T22:12:14.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 54ee31a7-a333-4fa1-b42f-903d8066c21d -i td/test-erasure-eio/0/new.json 2024-09-15T22:12:15.011 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:12:15.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T22:12:15.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+W+dm2m6yJRAASwXOFRCdorjcSgxF2k+ZMw== --osd-uuid 54ee31a7-a333-4fa1-b42f-903d8066c21d 2024-09-15T22:12:15.064 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:15.062+0000 7f8e1fb187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:15.076 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:15.074+0000 7f8e1fb187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:15.077 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:15.074+0000 7f8e1fb187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:15.078 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:15.078+0000 7f8e1fb187c0 -1 bdev(0x5632ef7c5c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:12:15.078 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:15.078+0000 7f8e1fb187c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T22:12:21.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T22:12:21.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:12:21.085 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T22:12:21.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:12:21.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:12:21.663 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T22:12:21.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:12:21.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:12:21.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:12:21.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:12:21.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:12:21.706 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:21.706+0000 7ff6ebc837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:21.713 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:21.710+0000 7ff6ebc837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:21.714 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:21.714+0000 7ff6ebc837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:22.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:12:22.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:12:22.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:12:22.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:12:22.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:12:22.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:22.107 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:12:22.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:12:22.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:22.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:22.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:23.161 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:23.158+0000 7ff6ebc837c0 -1 Falling back to public interface 2024-09-15T22:12:23.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:23.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:23.530 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:12:23.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:12:23.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:23.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:24.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:25.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:25.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:25.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:12:25.005 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:12:25.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:25.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:25.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:25.973 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:25.970+0000 7ff6ebc837c0 -1 osd.0 0 log_to_monitors true 2024-09-15T22:12:26.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:26.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:26.432 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:12:26.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:12:26.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:26.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:27.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:28.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:28.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:28.067 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:12:28.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:12:28.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:28.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:28.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:29.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:29.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:29.525 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:12:29.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:12:29.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:29.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:30.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:31.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:31.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:31.047 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:12:31.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:12:31.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:31.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:31.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:32.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:32.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:32.519 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:12:32.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:12:32.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:32.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:33.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:34.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:34.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:34.094 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:12:34.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:12:34.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:34.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:34.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:35.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:35.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:35.549 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:12:35.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:12:35.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:35.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:36.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:37.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:37.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:37.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:12:37.030 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:12:37.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:37.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:37.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:38.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:38.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:38.496 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:12:38.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:12:38.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:38.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:39.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:40.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:40.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:40.033 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:12:40.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:12:40.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:40.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:40.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:41.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:41.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:41.528 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:12:41.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:12:41.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:41.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:42.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:43.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:43.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:43.816 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:12:43.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:12:43.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:43.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:44.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:45.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:45.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:45.767 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:12:45.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:12:45.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:45.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:46.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:47.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:47.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:47.612 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:12:47.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:12:47.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:47.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:48.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:49.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:49.368 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:12:49.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:49.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:12:49.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:49.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:12:49.921 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3189307309,v1:127.0.0.1:6803/3189307309] [v2:127.0.0.1:6804/3189307309,v1:127.0.0.1:6805/3189307309] exists,up 54ee31a7-a333-4fa1-b42f-903d8066c21d 2024-09-15T22:12:49.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:12:49.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:12:49.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:12:49.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:12:49.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T22:12:49.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:12:49.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:12:49.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:12:49.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:12:49.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:12:49.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T22:12:49.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:12:49.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:12:49.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:12:49.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:12:49.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:12:49.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:12:49.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:12:49.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:12:49.926 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:12:49.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:12:49.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:12:49.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:12:49.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:12:49.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:12:49.928 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:12:49.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4a858d30-c74c-4a7d-b979-b096bf802cef 2024-09-15T22:12:49.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 4a858d30-c74c-4a7d-b979-b096bf802cef' 2024-09-15T22:12:49.929 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 4a858d30-c74c-4a7d-b979-b096bf802cef 2024-09-15T22:12:49.929 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:12:49.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDhW+dmQuXBOBAAYG2Av1FG3hOf3Pdqlea9wQ== 2024-09-15T22:12:49.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDhW+dmQuXBOBAAYG2Av1FG3hOf3Pdqlea9wQ=="}' 2024-09-15T22:12:49.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4a858d30-c74c-4a7d-b979-b096bf802cef -i td/test-erasure-eio/1/new.json 2024-09-15T22:12:50.592 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:12:50.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T22:12:50.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhW+dmQuXBOBAAYG2Av1FG3hOf3Pdqlea9wQ== --osd-uuid 4a858d30-c74c-4a7d-b979-b096bf802cef 2024-09-15T22:12:50.652 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:50.649+0000 7f166ac657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:50.654 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:50.649+0000 7f166ac657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:50.655 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:50.653+0000 7f166ac657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:50.655 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:50.653+0000 7f166ac657c0 -1 bdev(0x556870a97c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:12:50.656 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:50.653+0000 7f166ac657c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T22:12:56.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T22:12:56.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:12:56.969 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T22:12:56.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:12:56.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:12:57.589 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T22:12:57.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:12:57.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:12:57.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:12:57.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:12:57.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:12:57.629 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:57.625+0000 7fd66a3aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:57.637 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:57.633+0000 7fd66a3aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:57.639 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:57.637+0000 7fd66a3aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:12:58.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:12:58.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:12:58.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:12:58.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:12:58.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:12:58.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:58.101 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:12:58.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:12:58.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:12:58.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:58.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:12:58.905 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:12:58.901+0000 7fd66a3aa7c0 -1 Falling back to public interface 2024-09-15T22:12:59.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:12:59.560 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:12:59.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:12:59.561 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:12:59.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:12:59.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:00.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:01.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:01.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:01.019 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:13:01.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:13:01.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:01.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:01.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:01.720 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:13:01.717+0000 7fd66a3aa7c0 -1 osd.1 0 log_to_monitors true 2024-09-15T22:13:02.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:02.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:02.483 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:13:02.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:13:02.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:02.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:03.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:04.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:04.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:04.037 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:13:04.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:13:04.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:04.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:04.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:05.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:05.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:05.604 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:13:05.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:13:05.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:05.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:06.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:07.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:07.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:07.135 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:13:07.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:13:07.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:07.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:07.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:08.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:08.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:08.696 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:13:08.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:13:08.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:08.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:09.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:10.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:10.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:10.180 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:13:10.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:13:10.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:10.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:10.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:11.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:11.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:11.625 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:13:11.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:13:11.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:11.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:12.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:13.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:13.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:13.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:13:13.147 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:13:13.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:13.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:13.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:14.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:14.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:14.689 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:13:14.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:13:14.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:14.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:15.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:16.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:16.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:16.621 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:13:16.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:13:16.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:16.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:17.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:18.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:18.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:18.356 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:13:18.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:13:18.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:18.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:19.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:20.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:20.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:20.372 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:13:20.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:13:20.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:20.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:21.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:22.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:22.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:22.307 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:13:22.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:13:22.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:22.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:23.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:24.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:24.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:24.816 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:13:24.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:13:24.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:24.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:25.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:26.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:26.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:26.259 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:13:26.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:13:26.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:26.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:13:26.717 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2765447413,v1:127.0.0.1:6811/2765447413] [v2:127.0.0.1:6812/2765447413,v1:127.0.0.1:6813/2765447413] exists,up 4a858d30-c74c-4a7d-b979-b096bf802cef 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:13:26.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:13:26.722 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:13:26.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:13:26.724 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:13:26.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d07c1177-e502-4869-8ad8-498559151230 2024-09-15T22:13:26.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d07c1177-e502-4869-8ad8-498559151230' 2024-09-15T22:13:26.724 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 d07c1177-e502-4869-8ad8-498559151230 2024-09-15T22:13:26.725 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:13:26.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAGXOdm5B/GLBAAJGLHgqcnTLx3wmlYGqHXdg== 2024-09-15T22:13:26.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAGXOdm5B/GLBAAJGLHgqcnTLx3wmlYGqHXdg=="}' 2024-09-15T22:13:26.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d07c1177-e502-4869-8ad8-498559151230 -i td/test-erasure-eio/2/new.json 2024-09-15T22:13:27.234 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:13:27.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T22:13:27.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGXOdm5B/GLBAAJGLHgqcnTLx3wmlYGqHXdg== --osd-uuid d07c1177-e502-4869-8ad8-498559151230 2024-09-15T22:13:27.286 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:13:27.285+0000 7f0ed799f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:13:27.288 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:13:27.285+0000 7f0ed799f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:13:27.289 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:13:27.285+0000 7f0ed799f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:13:27.290 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:13:27.285+0000 7f0ed799f7c0 -1 bdev(0x564db27e1c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:13:27.290 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:13:27.285+0000 7f0ed799f7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T22:13:33.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T22:13:33.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:13:33.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:13:33.765 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T22:13:33.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:13:34.378 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:13:34.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:13:34.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:13:34.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:13:34.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:13:34.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:13:34.418 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:13:34.417+0000 7f690042f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:13:34.421 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:13:34.417+0000 7f690042f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:13:34.422 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:13:34.417+0000 7f690042f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:13:34.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:13:34.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:13:34.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:13:34.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:13:34.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:13:34.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:34.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:13:34.846 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:13:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:35.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:35.453 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:13:35.449+0000 7f690042f7c0 -1 Falling back to public interface 2024-09-15T22:13:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:36.270 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:13:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:13:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:36.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:37.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:37.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:37.743 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:13:37.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:13:37.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:37.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:38.129 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:13:38.125+0000 7f690042f7c0 -1 osd.2 0 log_to_monitors true 2024-09-15T22:13:38.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:39.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:39.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:39.220 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:13:39.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:13:39.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:39.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:39.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:40.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:40.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:40.719 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:13:40.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:13:40.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:40.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:41.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:42.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:42.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:42.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:13:42.222 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:13:42.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:42.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:42.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:43.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:43.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:43.666 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:13:43.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:13:43.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:43.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:44.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:45.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:45.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:45.284 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:13:45.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:13:45.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:45.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:45.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:46.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:46.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:46.721 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:13:46.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:13:46.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:46.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:47.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:48.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:48.184 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:13:48.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:13:48.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:48.185 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:48.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:49.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:49.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:49.715 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:13:49.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:13:49.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:49.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:50.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:51.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:51.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:51.322 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:13:51.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:13:51.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:51.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:51.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:52.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:52.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:52.937 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:13:52.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:13:52.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:52.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:53.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:54.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:54.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:54.916 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:13:54.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:13:54.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:54.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:55.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:56.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:56.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:56.663 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:13:56.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:13:56.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:56.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:57.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:13:58.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:13:58.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:13:58.477 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:13:58.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:13:58.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:13:58.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:13:59.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:00.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:00.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:00.226 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:14:00.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:14:00.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:00.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:14:01.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:02.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:02.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:02.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:14:02.183 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:14:02.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:02.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:14:02.799 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/794997453,v1:127.0.0.1:6819/794997453] [v2:127.0.0.1:6820/794997453,v1:127.0.0.1:6821/794997453] exists,up d07c1177-e502-4869-8ad8-498559151230 2024-09-15T22:14:02.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:14:02.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:14:02.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:14:02.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:14:02.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T22:14:02.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:14:02.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:14:02.802 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:14:02.803 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:14:02.803 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:14:02.803 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:14:02.803 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:14:02.803 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:14:02.804 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:14:02.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:14:02.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:14:02.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:14:02.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:14:02.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:14:02.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:14:02.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:14:02.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:14:02.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:14:02.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:14:02.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:14:02.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:14:02.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:14:02.805 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:14:02.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0b6cf7f5-8692-4654-93bf-6daed52f2faf 2024-09-15T22:14:02.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 0b6cf7f5-8692-4654-93bf-6daed52f2faf' 2024-09-15T22:14:02.806 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 0b6cf7f5-8692-4654-93bf-6daed52f2faf 2024-09-15T22:14:02.806 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:14:02.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAqXOdmzAsIMhAAlKFlbX1/XufXc3yTWI1w+Q== 2024-09-15T22:14:02.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAqXOdmzAsIMhAAlKFlbX1/XufXc3yTWI1w+Q=="}' 2024-09-15T22:14:02.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0b6cf7f5-8692-4654-93bf-6daed52f2faf -i td/test-erasure-eio/3/new.json 2024-09-15T22:14:03.943 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:14:03.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T22:14:03.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqXOdmzAsIMhAAlKFlbX1/XufXc3yTWI1w+Q== --osd-uuid 0b6cf7f5-8692-4654-93bf-6daed52f2faf 2024-09-15T22:14:04.009 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:04.004+0000 7f12d20777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:04.028 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:04.024+0000 7f12d20777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:04.030 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:04.024+0000 7f12d20777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:04.030 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:04.028+0000 7f12d20777c0 -1 bdev(0x55b511018800 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T22:14:04.030 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:04.028+0000 7f12d20777c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T22:14:10.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T22:14:10.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:14:10.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T22:14:10.612 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T22:14:10.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:14:11.212 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T22:14:11.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T22:14:11.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:14:11.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:14:11.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:14:11.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:14:11.253 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:11.248+0000 7fe9a9ed57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:11.256 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:11.252+0000 7fe9a9ed57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:11.257 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:11.252+0000 7fe9a9ed57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:11.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T22:14:11.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:14:11.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:14:11.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:14:11.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:14:11.681 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:14:11.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:11.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:14:11.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:11.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:12.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:12.537 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:12.532+0000 7fe9a9ed57c0 -1 Falling back to public interface 2024-09-15T22:14:13.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:13.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:13.147 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:14:13.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:14:13.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:13.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:13.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:14.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:14.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:14.584 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:14:14.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:14:14.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:14.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:15.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:15.302 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:15.296+0000 7fe9a9ed57c0 -1 osd.3 0 log_to_monitors true 2024-09-15T22:14:16.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:16.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:16.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:14:16.052 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:14:16.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:16.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:16.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:17.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:17.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:17.704 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:14:17.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:14:17.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:17.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:18.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:19.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:19.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:19.520 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:14:19.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:14:19.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:19.520 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:19.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:20.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:20.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:20.972 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:14:20.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:14:20.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:20.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:21.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:22.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:22.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:22.572 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:14:22.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:14:22.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:22.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:23.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:24.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:24.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:24.291 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:14:24.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:14:24.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:24.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:24.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:25.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:25.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:25.756 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:14:25.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:14:25.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:25.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:26.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:27.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:27.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:27.344 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:14:27.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:14:27.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:27.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:27.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:28.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:28.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:28.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:14:28.783 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:14:28.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:28.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:29.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:30.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:30.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:30.295 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:14:30.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:14:30.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:30.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:31.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:32.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:32.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:32.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:14:32.493 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:14:32.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:32.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:33.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:34.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:34.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:34.648 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:14:34.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:14:34.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:34.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:35.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:36.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:36.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:36.699 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:14:36.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:14:36.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:36.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:37.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:38.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:38.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:38.217 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:14:38.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:14:38.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:38.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:39.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:40.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:40.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:40.196 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:14:40.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:14:40.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:40.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:40.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:41.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:41.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:41.848 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:14:41.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:14:41.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:41.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:14:42.360 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1348593082,v1:127.0.0.1:6827/1348593082] [v2:127.0.0.1:6828/1348593082,v1:127.0.0.1:6829/1348593082] exists,up 0b6cf7f5-8692-4654-93bf-6daed52f2faf 2024-09-15T22:14:42.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:14:42.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:14:42.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:14:42.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:14:42.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T22:14:42.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:14:42.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:14:42.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-15T22:14:42.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:14:42.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T22:14:42.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T22:14:42.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:14:42.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:14:42.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:14:42.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T22:14:42.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T22:14:42.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:14:42.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:14:42.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:14:42.366 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:14:42.366 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:14:42.366 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:14:42.366 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:14:42.366 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:14:42.366 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:14:42.366 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:14:42.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:14:42.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T22:14:42.368 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:14:42.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=00ba925e-687f-42ba-92ca-61a46d770992 2024-09-15T22:14:42.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 00ba925e-687f-42ba-92ca-61a46d770992' 2024-09-15T22:14:42.368 INFO:tasks.workunit.client.0.smithi104.stdout:add osd4 00ba925e-687f-42ba-92ca-61a46d770992 2024-09-15T22:14:42.369 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:14:42.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBSXOdmwUefFxAAS/b5BBw97FmZvrxYG7A2Pg== 2024-09-15T22:14:42.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBSXOdmwUefFxAAS/b5BBw97FmZvrxYG7A2Pg=="}' 2024-09-15T22:14:42.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 00ba925e-687f-42ba-92ca-61a46d770992 -i td/test-erasure-eio/4/new.json 2024-09-15T22:14:43.669 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:14:43.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T22:14:43.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSXOdmwUefFxAAS/b5BBw97FmZvrxYG7A2Pg== --osd-uuid 00ba925e-687f-42ba-92ca-61a46d770992 2024-09-15T22:14:43.735 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:43.732+0000 7f8f75dfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:43.737 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:43.732+0000 7f8f75dfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:43.738 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:43.736+0000 7f8f75dfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:43.739 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:43.736+0000 7f8f75dfe7c0 -1 bdev(0x561e39a8bc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-15T22:14:43.739 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:43.736+0000 7f8f75dfe7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T22:14:50.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T22:14:50.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:14:50.125 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd4 key to auth repository 2024-09-15T22:14:50.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-15T22:14:50.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:14:50.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-15T22:14:50.746 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.4 2024-09-15T22:14:50.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:14:50.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:14:50.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:14:50.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:14:50.786 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:50.780+0000 7f77772cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:50.791 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:50.788+0000 7f77772cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:50.792 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:50.788+0000 7f77772cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:14:51.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-15T22:14:51.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:14:51.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-15T22:14:51.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:14:51.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:14:51.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:51.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:14:51.190 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:14:51.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:51.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:14:51.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:52.249 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:52.244+0000 7f77772cc7c0 -1 Falling back to public interface 2024-09-15T22:14:52.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:52.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:52.610 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:14:52.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:14:52.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:52.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:14:53.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:54.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:54.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:54.076 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:14:54.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:14:54.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:54.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:14:54.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:55.085 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:14:55.080+0000 7f77772cc7c0 -1 osd.4 0 log_to_monitors true 2024-09-15T22:14:55.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:55.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:55.581 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:14:55.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:14:55.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:55.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:14:56.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:57.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:57.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:57.289 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:14:57.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:14:57.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:57.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:14:58.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:14:59.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:14:59.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:14:59.200 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:14:59.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:14:59.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:14:59.200 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:14:59.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:00.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:00.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:00.684 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:15:00.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:15:00.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:00.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:01.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:02.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:02.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:02.145 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:15:02.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:15:02.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:02.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:02.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:03.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:03.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:03.640 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:15:03.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:15:03.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:03.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:04.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:05.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:05.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:05.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:15:05.323 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:15:05.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:05.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:05.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:06.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:06.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:06.928 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:15:06.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:15:06.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:06.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:07.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:08.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:08.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:08.424 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:15:08.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:15:08.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:08.424 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:08.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:09.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:09.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:09.961 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:15:09.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:15:09.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:09.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:11.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:12.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:12.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:12.385 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:15:12.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:15:12.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:12.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:13.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:14.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:14.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:14.308 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:15:14.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:15:14.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:14.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:15.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:16.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:16.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:16.422 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:15:16.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:15:16.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:16.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:17.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:18.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:18.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:18.309 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:15:18.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:15:18.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:18.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:19.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:15:20.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:15:20.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:15:20.289 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:15:20.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:15:20.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:15:20.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:15:21.079 INFO:tasks.workunit.client.0.smithi104.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2942308046,v1:127.0.0.1:6835/2942308046] [v2:127.0.0.1:6836/2942308046,v1:127.0.0.1:6837/2942308046] exists,up 00ba925e-687f-42ba-92ca-61a46d770992 2024-09-15T22:15:21.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:15:21.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:15:21.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:15:21.080 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:15:21.080 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:15:21.080 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:15:21.080 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:15:21.081 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:15:21.081 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:15:21.081 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:15:21.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:15:21.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:15:21.193 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:15:21.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:15:21.211 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:12:23.170+0000 7ff6ebc837c0 0 load: jerasure load: lrc 2024-09-15T22:15:21.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:15:21.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:641: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2024-09-15T22:15:21.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T22:15:21.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:15:21.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:15:21.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T22:15:21.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:15:21.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T22:15:21.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:15:21.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T22:15:21.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:15:21.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:15:23.388 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:15:23.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:15:24.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:15:24.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:15:24.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:15:24.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:15:24.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:15:24.409 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:15:24.409 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:15:24.410 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:15:24.410 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:15:24.410 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:15:24.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:15:24.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:15:24.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:15:24.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:15:24.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:15:24.535 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:15:25.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:15:25.037 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:15:25.037 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:15:25.037 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:15:25.037 INFO:tasks.workunit.client.0.smithi104.stderr:4' 2024-09-15T22:15:25.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:15:25.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:15:25.037 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:15:25.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803815 2024-09-15T22:15:25.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803815 2024-09-15T22:15:25.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803815' 2024-09-15T22:15:25.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:15:25.230 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:15:25.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607587 2024-09-15T22:15:25.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607587 2024-09-15T22:15:25.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803815 1-51539607587' 2024-09-15T22:15:25.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:15:25.385 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:15:25.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411351 2024-09-15T22:15:25.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411351 2024-09-15T22:15:25.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803815 1-51539607587 2-77309411351' 2024-09-15T22:15:25.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:15:25.558 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:15:25.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084300 2024-09-15T22:15:25.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084300 2024-09-15T22:15:25.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803815 1-51539607587 2-77309411351 3-120259084300' 2024-09-15T22:15:25.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:15:25.716 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T22:15:25.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691848 2024-09-15T22:15:25.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691848 2024-09-15T22:15:25.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803815 1-51539607587 2-77309411351 3-120259084300 4-171798691848' 2024-09-15T22:15:25.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:15:25.868 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803815 2024-09-15T22:15:25.868 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:15:25.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:15:25.869 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803815 2024-09-15T22:15:25.869 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:15:25.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803815 2024-09-15T22:15:25.870 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803815 2024-09-15T22:15:25.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2024-09-15T22:15:25.871 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:15:26.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803816 -lt 25769803815 2024-09-15T22:15:26.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:15:26.395 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607587 2024-09-15T22:15:26.395 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:15:26.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:15:26.397 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607587 2024-09-15T22:15:26.397 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:15:26.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607587 2024-09-15T22:15:26.400 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 51539607587 2024-09-15T22:15:26.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607587' 2024-09-15T22:15:26.400 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:15:26.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607587 -lt 51539607587 2024-09-15T22:15:26.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:15:26.826 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411351 2024-09-15T22:15:26.826 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:15:26.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:15:26.828 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411351 2024-09-15T22:15:26.828 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:15:26.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411351 2024-09-15T22:15:26.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411351' 2024-09-15T22:15:26.831 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 77309411351 2024-09-15T22:15:26.831 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:15:27.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411351 -lt 77309411351 2024-09-15T22:15:27.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:15:27.269 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084300 2024-09-15T22:15:27.269 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:15:27.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:15:27.272 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084300 2024-09-15T22:15:27.272 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:15:27.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084300 2024-09-15T22:15:27.273 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 120259084300 2024-09-15T22:15:27.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084300' 2024-09-15T22:15:27.274 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:15:27.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084300 -lt 120259084300 2024-09-15T22:15:27.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:15:27.698 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:15:27.698 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691848 2024-09-15T22:15:27.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T22:15:27.700 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691848 2024-09-15T22:15:27.700 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:15:27.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691848 2024-09-15T22:15:27.701 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 171798691848 2024-09-15T22:15:27.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691848' 2024-09-15T22:15:27.702 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T22:15:28.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691849 -lt 171798691848 2024-09-15T22:15:28.152 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:15:28.153 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:15:28.153 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:15:28.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:15:28.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:15:28.736 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:15:28.736 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:15:28.736 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:15:28.736 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:15:28.736 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:15:28.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:15:29.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:15:29.145 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:15:29.146 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:15:29.146 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:15:29.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:15:29.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:15:29.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:15:29.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:644: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-15T22:15:30.062 INFO:tasks.workunit.client.0.smithi104.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T22:15:30.062 INFO:tasks.workunit.client.0.smithi104.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T22:15:25.377904+0000 0'0 47:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-15T22:15:22.844531+0000 0'0 2024-09-15T22:15:22.844531+0000 0 0 periodic scrub scheduled @ 2024-09-17T01:48:39.887819+0000 0 0 2024-09-15T22:15:30.063 INFO:tasks.workunit.client.0.smithi104.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T22:15:23.986744+0000 0'0 46:15 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 periodic scrub scheduled @ 2024-09-17T09:44:44.297293+0000 0 0 2024-09-15T22:15:30.063 INFO:tasks.workunit.client.0.smithi104.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T22:15:23.986616+0000 0'0 46:109 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 periodic scrub scheduled @ 2024-09-17T01:24:40.393395+0000 0 0 2024-09-15T22:15:30.063 INFO:tasks.workunit.client.0.smithi104.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T22:15:23.864547+0000 0'0 46:15 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 periodic scrub scheduled @ 2024-09-17T07:08:25.528003+0000 0 0 2024-09-15T22:15:30.063 INFO:tasks.workunit.client.0.smithi104.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T22:14:03.223020+0000 0'0 46:74 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 periodic scrub scheduled @ 2024-09-17T03:34:50.087913+0000 0 0 2024-09-15T22:15:30.063 INFO:tasks.workunit.client.0.smithi104.stdout: 2024-09-15T22:15:30.063 INFO:tasks.workunit.client.0.smithi104.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T22:15:30.063 INFO:tasks.workunit.client.0.smithi104.stderr:dumped pgs 2024-09-15T22:15:30.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:15:30.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:30.260 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2024-09-15T22:15:30.261 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:15:30.261 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-15T22:15:30.261 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:15:30.261 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T22:15:30.704 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:15:30.704 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:15:30.704 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:15:30.704 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:15:30.704 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:15:30.705 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2 2024-09-15T22:15:30.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-09-15T22:15:30.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: local -a initial_osds 2024-09-15T22:15:30.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:649: TEST_ec_recovery_unfound: local last_osd=2 2024-09-15T22:15:30.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T22:15:30.705 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:15:30.705 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:15:30.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:15:30.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:15:30.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:15:31.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:15:31.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: ceph osd down 2 2024-09-15T22:15:31.848 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 is already down. 2024-09-15T22:15:31.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: ceph osd out 2 2024-09-15T22:15:32.716 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 is already out. 2024-09-15T22:15:32.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-15T22:15:33.151 INFO:tasks.workunit.client.0.smithi104.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T22:15:33.152 INFO:tasks.workunit.client.0.smithi104.stdout:2.0 1 0 1 0 0 4096 0 0 1 0 1 activating+undersized 2024-09-15T22:15:31.823948+0000 47'1 50:21 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-09-15T22:15:22.844531+0000 0'0 2024-09-15T22:15:22.844531+0000 0 0 no scrub is scheduled 0 0 2024-09-15T22:15:33.152 INFO:tasks.workunit.client.0.smithi104.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 peering 2024-09-15T22:15:31.136506+0000 0'0 48:19 [4,1] 4 [4,1] 4 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 no scrub is scheduled 0 0 2024-09-15T22:15:33.152 INFO:tasks.workunit.client.0.smithi104.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T22:15:23.986616+0000 0'0 47:111 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 periodic scrub scheduled @ 2024-09-17T01:24:40.393395+0000 0 0 2024-09-15T22:15:33.152 INFO:tasks.workunit.client.0.smithi104.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T22:15:23.864547+0000 0'0 49:21 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 periodic scrub scheduled @ 2024-09-17T07:08:25.528003+0000 0 0 2024-09-15T22:15:33.152 INFO:tasks.workunit.client.0.smithi104.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 peering 2024-09-15T22:15:30.980859+0000 0'0 47:77 [1,0] 1 [1,0] 1 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 no scrub is scheduled 0 0 2024-09-15T22:15:33.152 INFO:tasks.workunit.client.0.smithi104.stdout: 2024-09-15T22:15:33.152 INFO:tasks.workunit.client.0.smithi104.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T22:15:33.152 INFO:tasks.workunit.client.0.smithi104.stderr:dumped pgs 2024-09-15T22:15:33.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:656: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-15T22:15:33.171 INFO:tasks.workunit.client.0.smithi104.stderr:4+0 records in 2024-09-15T22:15:33.171 INFO:tasks.workunit.client.0.smithi104.stderr:4+0 records out 2024-09-15T22:15:33.171 INFO:tasks.workunit.client.0.smithi104.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000376832 s, 10.9 MB/s 2024-09-15T22:15:33.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: seq 1 100 2024-09-15T22:15:33.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:33.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:33.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:33.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:33.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:33.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:34.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:34.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:34.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:34.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:34.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:34.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:35.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:35.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:35.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:35.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:35.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:35.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:36.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:36.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:36.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:36.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:36.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:36.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:37.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:37.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:37.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:37.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:37.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:37.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:38.058 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:38.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:38.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:38.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:38.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:38.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:38.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:38.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:39.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:39.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:39.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:39.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:39.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:39.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:39.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:39.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:40.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:40.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:40.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:40.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:40.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:40.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:41.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:41.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:41.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:41.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:41.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:41.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:41.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:41.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:42.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:42.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:42.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:42.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:42.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:42.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:42.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:42.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:43.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:43.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:43.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:43.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:43.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:43.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:44.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:44.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:44.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:44.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:44.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:44.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:44.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:44.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:45.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:45.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:45.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:45.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:45.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:45.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:45.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:45.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:46.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:46.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:46.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:46.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:46.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:46.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:47.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:47.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:47.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:47.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:47.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:47.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:47.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:48.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:48.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:48.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:48.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:48.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:48.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:48.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:48.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:49.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:49.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:49.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:49.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:49.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:49.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:50.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:50.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:50.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:50.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:50.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:50.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:50.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:50.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:51.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:51.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:51.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:51.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:51.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:51.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:52.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:52.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:52.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:52.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:52.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:52.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:53.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:53.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:53.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:53.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:53.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:53.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:53.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:53.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:53.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:53.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:54.298 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:54.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:54.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:54.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:54.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:54.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:55.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:55.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:55.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:55.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:55.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:55.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:55.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:55.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:56.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:56.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:56.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:56.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:56.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:56.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:56.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:56.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:57.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:57.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:57.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:57.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:57.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:57.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:58.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:58.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:58.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:58.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:58.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:58.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:58.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:58.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:59.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:59.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:59.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:59.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:59.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:59.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-15T22:15:59.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:15:59.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-15T22:16:00.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:00.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-15T22:16:00.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:00.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-15T22:16:00.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj75 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T22:16:00.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:16:00.734 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj75 2024-09-15T22:16:00.734 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:16:00.734 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj75 2024-09-15T22:16:00.734 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-15T22:16:00.735 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:16:01.226 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:16:01.227 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:16:01.227 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:16:01.227 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:16:01.227 INFO:tasks.workunit.client.0.smithi104.stderr:2147483647' 2024-09-15T22:16:01.227 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-15T22:16:01.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-15T22:16:01.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:16:01.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-15T22:16:01.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:16:01.227 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T22:16:01.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:16:01.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T22:16:01.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:16:01.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-15T22:16:01.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:16:01.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:16:01.230 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:16:01.230 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-15T22:16:01.230 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:16:01.230 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:16:01.231 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:16:01.231 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:01.231 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:01.231 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:16:01.231 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:16:01.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:16:01.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:16:01.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-15T22:16:01.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:16:01.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:16:01.353 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:16:01.353 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:16:01.353 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:01.354 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:01.354 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:16:01.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:16:01.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2024-09-15T22:16:01.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:663: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2024-09-15T22:16:01.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:16:01.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:16:01.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:16:01.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:16:01.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:16:01.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:16:01.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj75 2024-09-15T22:16:01.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:16:01.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:16:01.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:16:01.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-15T22:16:01.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:16:01.474 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj75 2024-09-15T22:16:01.474 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:16:01.474 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj75 2024-09-15T22:16:01.474 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-15T22:16:01.474 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:16:01.943 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:16:01.943 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:16:01.943 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:16:01.943 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:16:01.943 INFO:tasks.workunit.client.0.smithi104.stderr:2147483647' 2024-09-15T22:16:01.943 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-15T22:16:01.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-15T22:16:01.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:16:01.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-15T22:16:01.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:16:01.945 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-15T22:16:01.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:16:01.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T22:16:01.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:16:01.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T22:16:01.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:16:01.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:16:01.946 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:16:01.946 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T22:16:01.946 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T22:16:01.946 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T22:16:01.947 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:16:01.947 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:01.947 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:01.947 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.1.asok 2024-09-15T22:16:01.947 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:16:02.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:16:02.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:16:02.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:16:02.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-15T22:16:02.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T22:16:02.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T22:16:02.058 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:16:02.058 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:02.058 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:02.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.1.asok 2024-09-15T22:16:02.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:16:02.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2024-09-15T22:16:02.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:665: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2024-09-15T22:16:02.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:16:02.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:16:02.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T22:16:02.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:16:02.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:16:02.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:16:02.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:16:02.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:16:02.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:16:02.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:16:02.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:16:02.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:16:02.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:16:02.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:16:02.169 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:16:02.169 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:16:02.169 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:16:02.169 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:16:02.169 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:02.169 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:02.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:16:02.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:16:02.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:16:02.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:16:02.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:16:02.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:16:02.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:16:02.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:16:02.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:16:02.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:16:02.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:16:02.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:16:02.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T22:16:02.173 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:16:02.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:16:02.174 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T22:16:02.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T22:16:02.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:16:02.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:16:02.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:16:02.201 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:02.199+0000 7f4cd2bde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:02.203 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:02.203+0000 7f4cd2bde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:02.205 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:02.203+0000 7f4cd2bde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:02.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T22:16:02.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:16:02.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:16:02.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:16:02.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:16:02.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:16:02.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:16:02.589 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:16:02.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:02.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:16:03.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:03.481 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:03.479+0000 7f4cd2bde7c0 -1 Falling back to public interface 2024-09-15T22:16:04.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:04.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:16:04.021 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:16:04.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:16:04.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:04.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:16:04.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:05.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:05.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:16:05.483 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:16:05.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:16:05.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:05.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:16:05.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:06.871 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:06.871+0000 7f4cd2bde7c0 -1 osd.2 47 log_to_monitors true 2024-09-15T22:16:06.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:06.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:16:06.909 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:16:06.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:16:06.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:06.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:16:07.488 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up out weight 0 up_from 55 up_thru 20 down_at 48 last_clean_interval [18,47) [v2:127.0.0.1:6818/15997049,v1:127.0.0.1:6819/15997049] [v2:127.0.0.1:6820/15997049,v1:127.0.0.1:6821/15997049] exists,up d07c1177-e502-4869-8ad8-498559151230 2024-09-15T22:16:07.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:16:07.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:16:07.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:16:07.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph osd in 2 2024-09-15T22:16:08.017 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 is already in. 2024-09-15T22:16:08.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:668: TEST_ec_recovery_unfound: sleep 15 2024-09-15T22:16:23.037 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: seq 1 100 2024-09-15T22:16:23.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2024-09-15T22:16:23.039 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:671: TEST_ec_recovery_unfound: get_state 2.0 2024-09-15T22:16:23.039 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T22:16:23.039 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T22:16:23.040 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T22:16:23.040 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T22:16:23.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:671: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2024-09-15T22:16:23.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:672: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2024-09-15T22:16:23.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:672: TEST_ec_recovery_unfound: grep recovery_unfound 2024-09-15T22:16:23.475 INFO:tasks.workunit.client.0.smithi104.stdout:active+recovery_unfound+degraded 2024-09-15T22:16:23.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2024-09-15T22:16:23.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: break 2024-09-15T22:16:23.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-15T22:16:23.882 INFO:tasks.workunit.client.0.smithi104.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T22:16:23.882 INFO:tasks.workunit.client.0.smithi104.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-09-15T22:16:12.106730+0000 53'101 57:347 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-15T22:15:22.844531+0000 0'0 2024-09-15T22:15:22.844531+0000 0 0 no scrub is scheduled 0 0 2024-09-15T22:16:23.882 INFO:tasks.workunit.client.0.smithi104.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T22:16:09.265485+0000 0'0 57:51 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 periodic scrub scheduled @ 2024-09-17T05:57:52.117609+0000 0 0 2024-09-15T22:16:23.882 INFO:tasks.workunit.client.0.smithi104.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T22:15:23.986616+0000 0'0 57:130 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 periodic scrub scheduled @ 2024-09-17T01:24:40.393395+0000 0 0 2024-09-15T22:16:23.882 INFO:tasks.workunit.client.0.smithi104.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T22:15:23.864547+0000 0'0 57:35 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 periodic scrub scheduled @ 2024-09-17T07:08:25.528003+0000 0 0 2024-09-15T22:16:23.882 INFO:tasks.workunit.client.0.smithi104.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T22:16:09.265088+0000 0'0 57:113 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T22:12:13.340894+0000 0'0 2024-09-15T22:12:13.340894+0000 0 0 periodic scrub scheduled @ 2024-09-17T02:25:44.084509+0000 0 0 2024-09-15T22:16:23.882 INFO:tasks.workunit.client.0.smithi104.stdout: 2024-09-15T22:16:23.882 INFO:tasks.workunit.client.0.smithi104.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T22:16:23.883 INFO:tasks.workunit.client.0.smithi104.stderr:dumped pgs 2024-09-15T22:16:23.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:681: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-15T22:16:24.042 INFO:tasks.workunit.client.0.smithi104.stdout:{ 2024-09-15T22:16:24.042 INFO:tasks.workunit.client.0.smithi104.stdout: "num_missing": 1, 2024-09-15T22:16:24.042 INFO:tasks.workunit.client.0.smithi104.stdout: "num_unfound": 1, 2024-09-15T22:16:24.042 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": [ 2024-09-15T22:16:24.042 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.042 INFO:tasks.workunit.client.0.smithi104.stdout: "oid": { 2024-09-15T22:16:24.042 INFO:tasks.workunit.client.0.smithi104.stdout: "oid": "obj75", 2024-09-15T22:16:24.042 INFO:tasks.workunit.client.0.smithi104.stdout: "key": "", 2024-09-15T22:16:24.042 INFO:tasks.workunit.client.0.smithi104.stdout: "snapid": -2, 2024-09-15T22:16:24.042 INFO:tasks.workunit.client.0.smithi104.stdout: "hash": 3565005477, 2024-09-15T22:16:24.042 INFO:tasks.workunit.client.0.smithi104.stdout: "max": 0, 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: "pool": 2, 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: "namespace": "" 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: "need": "53'76", 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: "have": "0'0", 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: "flags": "none", 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: "locations": [ 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: ] 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.043 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "NotRecovering", 2024-09-15T22:16:24.044 INFO:tasks.workunit.client.0.smithi104.stdout: "available_might_have_unfound": true, 2024-09-15T22:16:24.044 INFO:tasks.workunit.client.0.smithi104.stdout: "might_have_unfound": [], 2024-09-15T22:16:24.044 INFO:tasks.workunit.client.0.smithi104.stdout: "more": false 2024-09-15T22:16:24.044 INFO:tasks.workunit.client.0.smithi104.stdout:} 2024-09-15T22:16:24.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: ceph pg 2.0 query 2024-09-15T22:16:24.199 INFO:tasks.workunit.client.0.smithi104.stdout:{ 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: "snap_trimq": "[]", 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: "snap_trimq_len": 0, 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "active+recovery_unfound+degraded", 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch": 57, 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: "up": [ 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.200 INFO:tasks.workunit.client.0.smithi104.stdout: "acting": [ 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: "acting_recovery_backfill": [ 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: "1(1)", 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: "2(4)", 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: "3(0)", 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T22:16:24.201 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "info": { 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "pgid": "2.0s0", 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "shared": "0", 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "last_update": "53'101", 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "last_complete": "53'75", 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "log_tail": "0'0", 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "last_user_version": 101, 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "last_backfill": "MAX", 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [], 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "history": { 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_created": 45, 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_pool_created": 45, 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 57, 2024-09-15T22:16:24.202 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_started": 56, 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 46, 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_clean": 45, 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_split": 0, 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_marked_full": 0, 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "same_up_since": 56, 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "same_interval_since": 56, 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "same_primary_since": 45, 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.203 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "prior_readable_until_ub": 0 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "stats": { 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "version": "53'101", 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_seq": 348, 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_epoch": 57, 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "active+recovery_unfound+degraded", 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fresh": "2024-09-15T22:16:12.193063+0000", 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "last_change": "2024-09-15T22:16:12.106730+0000", 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "last_active": "2024-09-15T22:16:12.193063+0000", 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "last_peered": "2024-09-15T22:16:12.193063+0000", 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean": "2024-09-15T22:15:30.257239+0000", 2024-09-15T22:16:24.204 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_active": "2024-09-15T22:16:09.390295+0000", 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_peered": "2024-09-15T22:16:09.390295+0000", 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "last_unstale": "2024-09-15T22:16:12.193063+0000", 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "last_undegraded": "2024-09-15T22:16:09.006460+0000", 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fullsized": "2024-09-15T22:16:12.193063+0000", 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "mapping_epoch": 56, 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "log_start": "0'0", 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_start": "0'0", 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "created": 45, 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 46, 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "parent": "0.0", 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "parent_split_bits": 0, 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T22:16:24.205 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_scrubbed": 0, 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "log_size": 101, 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "log_dups_size": 0, 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_size": 101, 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "stats_invalid": false, 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "dirty_stats_invalid": false, 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "omap_stats_invalid": false, 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_stats_invalid": false, 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T22:16:24.206 INFO:tasks.workunit.client.0.smithi104.stdout: "pin_stats_invalid": false, 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "manifest_stats_invalid": false, 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrimq_len": 0, 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_duration": 0, 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_schedule": "no scrub is scheduled", 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_duration": 0, 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_trimmed": 0, 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrim_duration": 0, 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "stat_sum": { 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes": 413696, 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects": 101, 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_clones": 0, 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_copies": 505, 2024-09-15T22:16:24.207 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing_on_primary": 1, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing": 0, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_degraded": 3, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_misplaced": 0, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_unfound": 1, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_dirty": 101, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_whiteouts": 0, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read": 0, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read_kb": 0, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write": 101, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write_kb": 404, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_scrub_errors": 0, 2024-09-15T22:16:24.208 INFO:tasks.workunit.client.0.smithi104.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_deep_scrub_errors": 0, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_recovered": 99, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_recovered": 405504, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_keys_recovered": 0, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_omap": 0, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush": 0, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_kb": 0, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict": 0, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_kb": 0, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_promote": 0, 2024-09-15T22:16:24.209 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_high": 0, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_low": 0, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_some": 0, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_full": 0, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_pinned": 0, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: "num_legacy_snapsets": 0, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: "num_large_omap_objects": 0, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_manifest": 0, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_bytes": 0, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_keys": 0, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_repaired": 0 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: "up": [ 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.210 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: "acting": [ 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: "avail_no_missing": [ 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T22:16:24.211 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: "object_location_counts": [ 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": 100 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: "shards": "0(3),4(2)", 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": 1 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: "blocked_by": [], 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: "up_primary": 3, 2024-09-15T22:16:24.212 INFO:tasks.workunit.client.0.smithi104.stdout: "acting_primary": 3, 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [] 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: "empty": 0, 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: "dne": 0, 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: "incomplete": 0, 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 57, 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: "hit_set_history": { 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: "current_last_update": "0'0", 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: "history": [] 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: "peer_info": [ 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.213 INFO:tasks.workunit.client.0.smithi104.stdout: "peer": "0(3)", 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "pgid": "2.0s3", 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "shared": "3", 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "last_update": "53'101", 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "last_complete": "53'101", 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "log_tail": "0'0", 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "last_user_version": 101, 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "last_backfill": "MAX", 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [], 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "history": { 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_created": 45, 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_pool_created": 45, 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 57, 2024-09-15T22:16:24.214 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_started": 56, 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 46, 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_clean": 45, 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_split": 0, 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_marked_full": 0, 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "same_up_since": 56, 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "same_interval_since": 56, 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "same_primary_since": 45, 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.215 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "prior_readable_until_ub": 0 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "stats": { 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "version": "53'101", 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_seq": 135, 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_epoch": 53, 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "active+undersized+degraded", 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fresh": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "last_change": "2024-09-15T22:15:32.134981+0000", 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "last_active": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "last_peered": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean": "2024-09-15T22:15:30.257239+0000", 2024-09-15T22:16:24.216 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_active": "2024-09-15T22:15:32.134981+0000", 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_peered": "2024-09-15T22:15:32.134981+0000", 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "last_unstale": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "last_undegraded": "2024-09-15T22:15:31.823948+0000", 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fullsized": "2024-09-15T22:15:31.823641+0000", 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "mapping_epoch": 56, 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "log_start": "0'0", 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_start": "0'0", 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "created": 45, 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 46, 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "parent": "0.0", 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "parent_split_bits": 0, 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T22:16:24.217 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_scrubbed": 0, 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "log_size": 101, 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "log_dups_size": 0, 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_size": 101, 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "stats_invalid": false, 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "dirty_stats_invalid": false, 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "omap_stats_invalid": false, 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_stats_invalid": false, 2024-09-15T22:16:24.218 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "pin_stats_invalid": false, 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "manifest_stats_invalid": false, 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrimq_len": 0, 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_duration": 0, 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T07:38:14.553743+0000", 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_duration": 0, 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_trimmed": 0, 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrim_duration": 0, 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "stat_sum": { 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes": 413696, 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects": 101, 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_clones": 0, 2024-09-15T22:16:24.219 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_copies": 505, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing": 0, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_degraded": 101, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_misplaced": 0, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_unfound": 0, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_dirty": 101, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_whiteouts": 0, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read": 0, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read_kb": 0, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write": 101, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write_kb": 404, 2024-09-15T22:16:24.220 INFO:tasks.workunit.client.0.smithi104.stdout: "num_scrub_errors": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_deep_scrub_errors": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_recovered": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_recovered": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_keys_recovered": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_omap": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_kb": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_kb": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_promote": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_high": 0, 2024-09-15T22:16:24.221 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_low": 0, 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_some": 0, 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_full": 0, 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_pinned": 0, 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: "num_legacy_snapsets": 0, 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: "num_large_omap_objects": 0, 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_manifest": 0, 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_bytes": 0, 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_keys": 0, 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_repaired": 0 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: "up": [ 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.222 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: "acting": [ 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: "avail_no_missing": [ 2024-09-15T22:16:24.223 INFO:tasks.workunit.client.0.smithi104.stdout: "3(0)", 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: "1(1)", 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: "object_location_counts": [ 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": 101 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: "blocked_by": [], 2024-09-15T22:16:24.224 INFO:tasks.workunit.client.0.smithi104.stdout: "up_primary": 3, 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: "acting_primary": 3, 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [] 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: "empty": 0, 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: "dne": 0, 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: "incomplete": 0, 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 57, 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: "hit_set_history": { 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: "current_last_update": "0'0", 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: "history": [] 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.225 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "peer": "1(1)", 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "pgid": "2.0s1", 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "shared": "1", 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "last_update": "53'101", 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "last_complete": "53'101", 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "log_tail": "0'0", 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "last_user_version": 101, 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "last_backfill": "MAX", 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [], 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "history": { 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_created": 45, 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_pool_created": 45, 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 57, 2024-09-15T22:16:24.226 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_started": 56, 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 46, 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_clean": 45, 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_split": 0, 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_marked_full": 0, 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "same_up_since": 56, 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "same_interval_since": 56, 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "same_primary_since": 45, 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: "prior_readable_until_ub": 0 2024-09-15T22:16:24.227 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.228 INFO:tasks.workunit.client.0.smithi104.stdout: "stats": { 2024-09-15T22:16:24.228 INFO:tasks.workunit.client.0.smithi104.stdout: "version": "53'101", 2024-09-15T22:16:24.228 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_seq": 135, 2024-09-15T22:16:24.228 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_epoch": 53, 2024-09-15T22:16:24.228 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "active+undersized+degraded", 2024-09-15T22:16:24.228 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fresh": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.228 INFO:tasks.workunit.client.0.smithi104.stdout: "last_change": "2024-09-15T22:15:32.134981+0000", 2024-09-15T22:16:24.228 INFO:tasks.workunit.client.0.smithi104.stdout: "last_active": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.228 INFO:tasks.workunit.client.0.smithi104.stdout: "last_peered": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.228 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean": "2024-09-15T22:15:30.257239+0000", 2024-09-15T22:16:24.228 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_active": "2024-09-15T22:15:32.134981+0000", 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_peered": "2024-09-15T22:15:32.134981+0000", 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "last_unstale": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "last_undegraded": "2024-09-15T22:15:31.823948+0000", 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fullsized": "2024-09-15T22:15:31.823641+0000", 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "mapping_epoch": 56, 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "log_start": "0'0", 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_start": "0'0", 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "created": 45, 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 46, 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "parent": "0.0", 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "parent_split_bits": 0, 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T22:16:24.229 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.230 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.230 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_scrubbed": 0, 2024-09-15T22:16:24.230 INFO:tasks.workunit.client.0.smithi104.stdout: "log_size": 101, 2024-09-15T22:16:24.230 INFO:tasks.workunit.client.0.smithi104.stdout: "log_dups_size": 0, 2024-09-15T22:16:24.230 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_size": 101, 2024-09-15T22:16:24.230 INFO:tasks.workunit.client.0.smithi104.stdout: "stats_invalid": false, 2024-09-15T22:16:24.230 INFO:tasks.workunit.client.0.smithi104.stdout: "dirty_stats_invalid": false, 2024-09-15T22:16:24.230 INFO:tasks.workunit.client.0.smithi104.stdout: "omap_stats_invalid": false, 2024-09-15T22:16:24.230 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_stats_invalid": false, 2024-09-15T22:16:24.230 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T22:16:24.230 INFO:tasks.workunit.client.0.smithi104.stdout: "pin_stats_invalid": false, 2024-09-15T22:16:24.231 INFO:tasks.workunit.client.0.smithi104.stdout: "manifest_stats_invalid": false, 2024-09-15T22:16:24.231 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrimq_len": 0, 2024-09-15T22:16:24.231 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_duration": 0, 2024-09-15T22:16:24.231 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T07:38:14.553743+0000", 2024-09-15T22:16:24.231 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_duration": 0, 2024-09-15T22:16:24.231 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_trimmed": 0, 2024-09-15T22:16:24.231 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrim_duration": 0, 2024-09-15T22:16:24.231 INFO:tasks.workunit.client.0.smithi104.stdout: "stat_sum": { 2024-09-15T22:16:24.231 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes": 413696, 2024-09-15T22:16:24.231 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects": 101, 2024-09-15T22:16:24.231 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_clones": 0, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_copies": 505, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing": 1, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_degraded": 101, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_misplaced": 0, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_unfound": 0, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_dirty": 101, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_whiteouts": 0, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read": 0, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read_kb": 0, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write": 101, 2024-09-15T22:16:24.232 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write_kb": 404, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_scrub_errors": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_deep_scrub_errors": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_recovered": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_recovered": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_keys_recovered": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_omap": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_kb": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_kb": 0, 2024-09-15T22:16:24.233 INFO:tasks.workunit.client.0.smithi104.stdout: "num_promote": 0, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_high": 0, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_low": 0, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_some": 0, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_full": 0, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_pinned": 0, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "num_legacy_snapsets": 0, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "num_large_omap_objects": 0, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_manifest": 0, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_bytes": 0, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_keys": 0, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_repaired": 0 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.234 INFO:tasks.workunit.client.0.smithi104.stdout: "up": [ 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: "acting": [ 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.235 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.236 INFO:tasks.workunit.client.0.smithi104.stdout: "avail_no_missing": [ 2024-09-15T22:16:24.236 INFO:tasks.workunit.client.0.smithi104.stdout: "3(0)", 2024-09-15T22:16:24.236 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T22:16:24.236 INFO:tasks.workunit.client.0.smithi104.stdout: "1(1)", 2024-09-15T22:16:24.236 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T22:16:24.236 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.236 INFO:tasks.workunit.client.0.smithi104.stdout: "object_location_counts": [ 2024-09-15T22:16:24.236 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.236 INFO:tasks.workunit.client.0.smithi104.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T22:16:24.236 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": 101 2024-09-15T22:16:24.236 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: "blocked_by": [], 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: "up_primary": 3, 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: "acting_primary": 3, 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [] 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: "empty": 0, 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: "dne": 0, 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: "incomplete": 0, 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 57, 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: "hit_set_history": { 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: "current_last_update": "0'0", 2024-09-15T22:16:24.237 INFO:tasks.workunit.client.0.smithi104.stdout: "history": [] 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: "peer": "2(4)", 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: "pgid": "2.0s4", 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: "shared": "4", 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: "last_update": "53'101", 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: "last_complete": "47'1", 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: "log_tail": "0'0", 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: "last_user_version": 1, 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: "last_backfill": "MAX", 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [], 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: "history": { 2024-09-15T22:16:24.238 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_created": 45, 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_pool_created": 45, 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 57, 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_started": 56, 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 46, 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_clean": 45, 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_split": 0, 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_marked_full": 0, 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "same_up_since": 56, 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "same_interval_since": 56, 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "same_primary_since": 45, 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T22:16:24.239 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "prior_readable_until_ub": 0 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "stats": { 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "version": "47'1", 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_seq": 18, 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_epoch": 47, 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "active+clean", 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fresh": "2024-09-15T22:15:30.257239+0000", 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "last_change": "2024-09-15T22:15:25.377904+0000", 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "last_active": "2024-09-15T22:15:30.257239+0000", 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "last_peered": "2024-09-15T22:15:30.257239+0000", 2024-09-15T22:16:24.240 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean": "2024-09-15T22:15:30.257239+0000", 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_active": "2024-09-15T22:15:25.376741+0000", 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_peered": "2024-09-15T22:15:25.376741+0000", 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "last_unstale": "2024-09-15T22:15:30.257239+0000", 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "last_undegraded": "2024-09-15T22:15:30.257239+0000", 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fullsized": "2024-09-15T22:15:30.257239+0000", 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "mapping_epoch": 56, 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "log_start": "0'0", 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_start": "0'0", 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "created": 45, 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 46, 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "parent": "0.0", 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "parent_split_bits": 0, 2024-09-15T22:16:24.241 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_scrubbed": 0, 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "log_size": 1, 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "log_dups_size": 0, 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_size": 1, 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "stats_invalid": false, 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "dirty_stats_invalid": false, 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "omap_stats_invalid": false, 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_stats_invalid": false, 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "pin_stats_invalid": false, 2024-09-15T22:16:24.242 INFO:tasks.workunit.client.0.smithi104.stdout: "manifest_stats_invalid": false, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrimq_len": 0, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_duration": 0, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T01:48:39.887819+0000", 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_duration": 0, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_trimmed": 0, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrim_duration": 0, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "stat_sum": { 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes": 4096, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects": 1, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_clones": 0, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_copies": 5, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing": 1, 2024-09-15T22:16:24.243 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_degraded": 0, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_misplaced": 0, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_unfound": 0, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_dirty": 1, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_whiteouts": 0, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read": 0, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read_kb": 0, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write": 1, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write_kb": 4, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_scrub_errors": 0, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_deep_scrub_errors": 0, 2024-09-15T22:16:24.244 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_recovered": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_recovered": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_keys_recovered": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_omap": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_kb": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_kb": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_promote": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_high": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_low": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_some": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_full": 0, 2024-09-15T22:16:24.245 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_pinned": 0, 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: "num_legacy_snapsets": 0, 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: "num_large_omap_objects": 0, 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_manifest": 0, 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_bytes": 0, 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_keys": 0, 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_repaired": 0 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: "up": [ 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.246 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: "acting": [ 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: "avail_no_missing": [], 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: "object_location_counts": [], 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: "blocked_by": [], 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: "up_primary": 3, 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: "acting_primary": 3, 2024-09-15T22:16:24.247 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [] 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: "empty": 0, 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: "dne": 0, 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: "incomplete": 0, 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 57, 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: "hit_set_history": { 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: "current_last_update": "0'0", 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: "history": [] 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: "peer": "4(2)", 2024-09-15T22:16:24.248 INFO:tasks.workunit.client.0.smithi104.stdout: "pgid": "2.0s2", 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "shared": "2", 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "last_update": "53'101", 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "last_complete": "53'101", 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "log_tail": "0'0", 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "last_user_version": 101, 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "last_backfill": "MAX", 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [], 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "history": { 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_created": 45, 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "epoch_pool_created": 45, 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 57, 2024-09-15T22:16:24.249 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_started": 56, 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 46, 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "last_interval_clean": 45, 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_split": 0, 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_marked_full": 0, 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "same_up_since": 56, 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "same_interval_since": 56, 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "same_primary_since": 45, 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.250 INFO:tasks.workunit.client.0.smithi104.stdout: "prior_readable_until_ub": 0 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "stats": { 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "version": "53'101", 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_seq": 135, 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "reported_epoch": 53, 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "state": "active+undersized+degraded", 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fresh": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "last_change": "2024-09-15T22:15:32.134981+0000", 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "last_active": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "last_peered": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean": "2024-09-15T22:15:30.257239+0000", 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_active": "2024-09-15T22:15:32.134981+0000", 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "last_became_peered": "2024-09-15T22:15:32.134981+0000", 2024-09-15T22:16:24.251 INFO:tasks.workunit.client.0.smithi104.stdout: "last_unstale": "2024-09-15T22:16:00.729235+0000", 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "last_undegraded": "2024-09-15T22:15:31.823948+0000", 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "last_fullsized": "2024-09-15T22:15:31.823641+0000", 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "mapping_epoch": 56, 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "log_start": "0'0", 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_start": "0'0", 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "created": 45, 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_clean": 46, 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "parent": "0.0", 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "parent_split_bits": 0, 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub": "0'0", 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub": "0'0", 2024-09-15T22:16:24.252 INFO:tasks.workunit.client.0.smithi104.stdout: "last_deep_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "last_clean_scrub_stamp": "2024-09-15T22:15:22.844531+0000", 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_scrubbed": 0, 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "log_size": 101, 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "log_dups_size": 0, 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "ondisk_log_size": 101, 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "stats_invalid": false, 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "dirty_stats_invalid": false, 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "omap_stats_invalid": false, 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_stats_invalid": false, 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "pin_stats_invalid": false, 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "manifest_stats_invalid": false, 2024-09-15T22:16:24.253 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrimq_len": 0, 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "last_scrub_duration": 0, 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T07:38:14.553743+0000", 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_duration": 0, 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "objects_trimmed": 0, 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "snaptrim_duration": 0, 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "stat_sum": { 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes": 413696, 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects": 101, 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_clones": 0, 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "num_object_copies": 505, 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T22:16:24.254 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_missing": 0, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_degraded": 101, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_misplaced": 0, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_unfound": 0, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_dirty": 101, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_whiteouts": 0, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read": 0, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_read_kb": 0, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write": 101, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_write_kb": 404, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_scrub_errors": 0, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_deep_scrub_errors": 0, 2024-09-15T22:16:24.255 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_recovered": 0, 2024-09-15T22:16:24.256 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_recovered": 0, 2024-09-15T22:16:24.256 INFO:tasks.workunit.client.0.smithi104.stdout: "num_keys_recovered": 0, 2024-09-15T22:16:24.256 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_omap": 0, 2024-09-15T22:16:24.256 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T22:16:24.256 INFO:tasks.workunit.client.0.smithi104.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T22:16:24.256 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush": 0, 2024-09-15T22:16:24.256 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_kb": 0, 2024-09-15T22:16:24.256 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict": 0, 2024-09-15T22:16:24.256 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_kb": 0, 2024-09-15T22:16:24.256 INFO:tasks.workunit.client.0.smithi104.stdout: "num_promote": 0, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_high": 0, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "num_flush_mode_low": 0, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_some": 0, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "num_evict_mode_full": 0, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_pinned": 0, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "num_legacy_snapsets": 0, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "num_large_omap_objects": 0, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_manifest": 0, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_bytes": 0, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "num_omap_keys": 0, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "num_objects_repaired": 0 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.257 INFO:tasks.workunit.client.0.smithi104.stdout: "up": [ 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: "acting": [ 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: 3, 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: 1, 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: 4, 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: 0, 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: 2 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: "avail_no_missing": [ 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: "3(0)", 2024-09-15T22:16:24.258 INFO:tasks.workunit.client.0.smithi104.stdout: "0(3)", 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: "1(1)", 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: "4(2)" 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: "object_location_counts": [ 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": 101 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: "blocked_by": [], 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: "up_primary": 3, 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: "acting_primary": 3, 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: "purged_snaps": [] 2024-09-15T22:16:24.259 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: "empty": 0, 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: "dne": 0, 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: "incomplete": 0, 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: "last_epoch_started": 57, 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: "hit_set_history": { 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: "current_last_update": "0'0", 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: "history": [] 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: "recovery_state": [ 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: "name": "Started/Primary/Active", 2024-09-15T22:16:24.260 INFO:tasks.workunit.client.0.smithi104.stdout: "enter_time": "2024-09-15T22:16:09.005065+0000", 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: "might_have_unfound": [ 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: "osd": "0(3)", 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: "status": "already probed" 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: "osd": "1(1)", 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: "status": "already probed" 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: "osd": "2(4)", 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: "status": "already probed" 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.261 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: "osd": "4(2)", 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: "status": "already probed" 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: "recovery_progress": { 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: "backfill_targets": [], 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: "waiting_on_backfill": [], 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: "last_backfill_started": "MIN", 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: "backfill_info": { 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: "begin": "MIN", 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: "end": "MIN", 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: "objects": [] 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.262 INFO:tasks.workunit.client.0.smithi104.stdout: "peer_backfill_info": [], 2024-09-15T22:16:24.263 INFO:tasks.workunit.client.0.smithi104.stdout: "backfills_in_flight": [], 2024-09-15T22:16:24.263 INFO:tasks.workunit.client.0.smithi104.stdout: "recovering": [], 2024-09-15T22:16:24.263 INFO:tasks.workunit.client.0.smithi104.stdout: "pg_backend": { 2024-09-15T22:16:24.263 INFO:tasks.workunit.client.0.smithi104.stdout: "recovery_ops": [], 2024-09-15T22:16:24.263 INFO:tasks.workunit.client.0.smithi104.stdout: "read_ops": [] 2024-09-15T22:16:24.263 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.263 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.263 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.263 INFO:tasks.workunit.client.0.smithi104.stdout: { 2024-09-15T22:16:24.263 INFO:tasks.workunit.client.0.smithi104.stdout: "name": "Started", 2024-09-15T22:16:24.263 INFO:tasks.workunit.client.0.smithi104.stdout: "enter_time": "2024-09-15T22:16:08.134671+0000" 2024-09-15T22:16:24.264 INFO:tasks.workunit.client.0.smithi104.stdout: } 2024-09-15T22:16:24.264 INFO:tasks.workunit.client.0.smithi104.stdout: ], 2024-09-15T22:16:24.264 INFO:tasks.workunit.client.0.smithi104.stdout: "scrubber": { 2024-09-15T22:16:24.264 INFO:tasks.workunit.client.0.smithi104.stdout: "active": false, 2024-09-15T22:16:24.264 INFO:tasks.workunit.client.0.smithi104.stdout: "must_scrub": false, 2024-09-15T22:16:24.264 INFO:tasks.workunit.client.0.smithi104.stdout: "must_deep_scrub": false, 2024-09-15T22:16:24.264 INFO:tasks.workunit.client.0.smithi104.stdout: "must_repair": false, 2024-09-15T22:16:24.264 INFO:tasks.workunit.client.0.smithi104.stdout: "need_auto": false, 2024-09-15T22:16:24.264 INFO:tasks.workunit.client.0.smithi104.stdout: "scrub_reg_stamp": "2024-09-17T07:38:14.553743+0000", 2024-09-15T22:16:24.264 INFO:tasks.workunit.client.0.smithi104.stdout: "schedule": "not registered for scrubbing" 2024-09-15T22:16:24.264 INFO:tasks.workunit.client.0.smithi104.stdout: }, 2024-09-15T22:16:24.265 INFO:tasks.workunit.client.0.smithi104.stdout: "agent_state": {} 2024-09-15T22:16:24.265 INFO:tasks.workunit.client.0.smithi104.stdout:} 2024-09-15T22:16:24.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-15T22:16:24.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: grep -q obj75 2024-09-15T22:16:24.400 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-15T22:16:24.400 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2024-09-15T22:16:24.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: check=true 2024-09-15T22:16:24.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: test true == true 2024-09-15T22:16:24.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-15T22:16:24.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2024-09-15T22:16:24.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: check=0 2024-09-15T22:16:24.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:690: TEST_ec_recovery_unfound: test 0 == 0 2024-09-15T22:16:24.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-15T22:16:29.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:694: TEST_ec_recovery_unfound: test 124 = 124 2024-09-15T22:16:29.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:696: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-15T22:16:30.000 INFO:tasks.workunit.client.0.smithi104.stderr:pg has no unfound objects 2024-09-15T22:16:30.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:698: TEST_ec_recovery_unfound: wait_for_clean 2024-09-15T22:16:30.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:16:30.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:16:30.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:16:30.021 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:16:30.022 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:16:30.022 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:16:30.022 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:16:30.023 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:16:30.023 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:16:30.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:16:30.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:16:30.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:16:30.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:16:30.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:16:30.139 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:16:30.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:16:30.694 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:16:30.694 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:16:30.694 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:16:30.694 INFO:tasks.workunit.client.0.smithi104.stderr:4' 2024-09-15T22:16:30.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:16:30.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:16:30.695 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:16:30.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803831 2024-09-15T22:16:30.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803831 2024-09-15T22:16:30.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831' 2024-09-15T22:16:30.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:16:30.877 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:16:31.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607602 2024-09-15T22:16:31.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607602 2024-09-15T22:16:31.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-51539607602' 2024-09-15T22:16:31.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:16:31.030 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:16:31.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201287 2024-09-15T22:16:31.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201287 2024-09-15T22:16:31.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-51539607602 2-236223201287' 2024-09-15T22:16:31.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:16:31.182 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:16:31.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084315 2024-09-15T22:16:31.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084315 2024-09-15T22:16:31.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-51539607602 2-236223201287 3-120259084315' 2024-09-15T22:16:31.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:16:31.338 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T22:16:31.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691863 2024-09-15T22:16:31.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691863 2024-09-15T22:16:31.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-51539607602 2-236223201287 3-120259084315 4-171798691863' 2024-09-15T22:16:31.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:16:31.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803831 2024-09-15T22:16:31.511 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:16:31.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:16:31.513 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803831 2024-09-15T22:16:31.513 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:16:31.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803831 2024-09-15T22:16:31.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-09-15T22:16:31.514 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803831 2024-09-15T22:16:31.514 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:16:31.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803828 -lt 25769803831 2024-09-15T22:16:31.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:16:32.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:16:32.917 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:16:33.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803831 -lt 25769803831 2024-09-15T22:16:33.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:16:33.380 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607602 2024-09-15T22:16:33.380 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:16:33.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:16:33.383 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607602 2024-09-15T22:16:33.383 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:16:33.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607602 2024-09-15T22:16:33.385 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 51539607602 2024-09-15T22:16:33.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607602' 2024-09-15T22:16:33.385 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:16:33.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607603 -lt 51539607602 2024-09-15T22:16:33.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:16:33.901 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-236223201287 2024-09-15T22:16:33.901 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:16:33.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:16:33.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-236223201287 2024-09-15T22:16:33.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:16:33.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201287 2024-09-15T22:16:33.905 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 236223201287 2024-09-15T22:16:33.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 236223201287' 2024-09-15T22:16:33.906 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:16:34.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201288 -lt 236223201287 2024-09-15T22:16:34.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:16:34.374 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084315 2024-09-15T22:16:34.374 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:16:34.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:16:34.376 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084315 2024-09-15T22:16:34.376 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:16:34.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084315 2024-09-15T22:16:34.378 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 120259084315 2024-09-15T22:16:34.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084315' 2024-09-15T22:16:34.378 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:16:34.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084316 -lt 120259084315 2024-09-15T22:16:34.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:16:34.886 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691863 2024-09-15T22:16:34.886 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:16:34.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T22:16:34.888 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691863 2024-09-15T22:16:34.888 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:16:34.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691863 2024-09-15T22:16:34.890 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 171798691863 2024-09-15T22:16:34.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691863' 2024-09-15T22:16:34.890 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T22:16:35.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691864 -lt 171798691863 2024-09-15T22:16:35.362 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:16:35.362 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:16:35.362 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:16:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:16:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:16:35.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:16:35.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:16:35.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:16:35.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:16:35.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:16:35.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:16:36.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:16:36.400 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:16:36.400 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:16:36.400 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:16:36.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:16:36.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:16:36.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:16:36.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: seq 1 100 2024-09-15T22:16:36.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:36.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2024-09-15T22:16:36.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-15T22:16:36.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:36.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:36.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2024-09-15T22:16:36.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2024-09-15T22:16:37.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2024-09-15T22:16:37.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2024-09-15T22:16:37.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.158 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2024-09-15T22:16:37.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2024-09-15T22:16:37.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2024-09-15T22:16:37.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2024-09-15T22:16:37.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2024-09-15T22:16:37.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2024-09-15T22:16:37.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2024-09-15T22:16:37.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2024-09-15T22:16:37.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2024-09-15T22:16:37.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2024-09-15T22:16:37.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2024-09-15T22:16:37.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2024-09-15T22:16:37.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2024-09-15T22:16:37.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2024-09-15T22:16:37.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2024-09-15T22:16:37.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2024-09-15T22:16:37.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2024-09-15T22:16:37.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2024-09-15T22:16:37.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2024-09-15T22:16:37.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2024-09-15T22:16:37.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2024-09-15T22:16:37.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2024-09-15T22:16:37.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2024-09-15T22:16:37.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2024-09-15T22:16:37.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2024-09-15T22:16:37.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-15T22:16:37.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:37.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:37.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2024-09-15T22:16:37.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2024-09-15T22:16:38.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2024-09-15T22:16:38.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2024-09-15T22:16:38.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2024-09-15T22:16:38.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2024-09-15T22:16:38.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2024-09-15T22:16:38.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2024-09-15T22:16:38.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2024-09-15T22:16:38.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2024-09-15T22:16:38.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2024-09-15T22:16:38.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2024-09-15T22:16:38.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2024-09-15T22:16:38.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2024-09-15T22:16:38.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2024-09-15T22:16:38.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2024-09-15T22:16:38.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.571 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2024-09-15T22:16:38.572 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2024-09-15T22:16:38.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2024-09-15T22:16:38.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2024-09-15T22:16:38.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2024-09-15T22:16:38.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2024-09-15T22:16:38.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2024-09-15T22:16:38.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2024-09-15T22:16:38.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2024-09-15T22:16:38.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2024-09-15T22:16:38.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2024-09-15T22:16:38.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2024-09-15T22:16:38.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2024-09-15T22:16:38.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2024-09-15T22:16:38.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-15T22:16:38.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:38.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:38.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2024-09-15T22:16:38.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2024-09-15T22:16:39.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.044 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2024-09-15T22:16:39.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2024-09-15T22:16:39.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2024-09-15T22:16:39.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2024-09-15T22:16:39.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2024-09-15T22:16:39.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2024-09-15T22:16:39.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2024-09-15T22:16:39.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2024-09-15T22:16:39.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2024-09-15T22:16:39.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2024-09-15T22:16:39.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2024-09-15T22:16:39.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2024-09-15T22:16:39.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2024-09-15T22:16:39.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:704: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.426 INFO:tasks.workunit.client.0.smithi104.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2024-09-15T22:16:39.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2024-09-15T22:16:39.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.457 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2024-09-15T22:16:39.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2024-09-15T22:16:39.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2024-09-15T22:16:39.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2024-09-15T22:16:39.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2024-09-15T22:16:39.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2024-09-15T22:16:39.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2024-09-15T22:16:39.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2024-09-15T22:16:39.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2024-09-15T22:16:39.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2024-09-15T22:16:39.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2024-09-15T22:16:39.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2024-09-15T22:16:39.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2024-09-15T22:16:39.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2024-09-15T22:16:39.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2024-09-15T22:16:39.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2024-09-15T22:16:39.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2024-09-15T22:16:39.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-15T22:16:39.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:39.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:39.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2024-09-15T22:16:39.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-15T22:16:40.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:40.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:40.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2024-09-15T22:16:40.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-15T22:16:40.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:40.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:40.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2024-09-15T22:16:40.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-15T22:16:40.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:40.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:40.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2024-09-15T22:16:40.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-15T22:16:40.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:40.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:40.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2024-09-15T22:16:40.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-15T22:16:40.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:40.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:40.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2024-09-15T22:16:40.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-15T22:16:40.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:40.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T22:16:40.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2024-09-15T22:16:40.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-15T22:16:40.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:40.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:711: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T22:16:40.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:713: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:16:40.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:16:40.199 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:16:40.692 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:16:40.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:16:41.435 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:16:41.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:16:41.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:16:41.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:16:41.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:16:41.455 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:16:41.455 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:16:41.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:16:41.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:16:41.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:16:41.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:16:41.769 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:16:41.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:16:41.770 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:16:41.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:16:41.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:16:41.771 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:16:41.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:16:41.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:16:41.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:16:41.773 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:16:41.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:16:41.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:16:41.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:16:41.816 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:16:41.816 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:41.816 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:41.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:16:41.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:16:41.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:16:41.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:16:41.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:16:41.821 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:16:41.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:16:41.823 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:16:41.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:16:41.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:16:41.824 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:16:41.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:16:41.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:16:41.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:16:41.826 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:16:41.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:16:41.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:16:41.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:16:41.829 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:16:41.829 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:41.829 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:41.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:16:41.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:16:41.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:16:41.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T22:16:41.832 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:16:41.832 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:41.832 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:41.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T22:16:41.834 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:16:41.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:16:41.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:16:41.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T22:16:41.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T22:16:41.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T22:16:41.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:16:41.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:16:41.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:16:41.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T22:16:41.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T22:16:42.304 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:16:42.304 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:16:42.304 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:16:42.305 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:16:42.305 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:42.305 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:42.305 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:16:42.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T22:16:42.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:16:42.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:16:42.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:16:42.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:16:42.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:16:42.519 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:16:42.519 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:16:42.519 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:16:42.519 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:16:42.519 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:16:42.519 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:42.519 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:42.520 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:16:42.520 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:16:42.520 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T22:16:42.638 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:16:42.638 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:16:42.638 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:16:42.638 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:16:42.639 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:16:42.639 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:16:42.639 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:16:42.639 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:16:42.639 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:16:42.639 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:42.640 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:42.640 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:16:42.640 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:16:42.640 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T22:16:42.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T22:16:42.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T22:16:42.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:16:42.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:16:42.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:16:42.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T22:16:42.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:16:43.653 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:16:43.653 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:16:43.653 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:16:43.654 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:16:43.654 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:43.654 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:43.654 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:16:43.654 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:16:43.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:16:43.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T22:16:43.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T22:16:44.002 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T22:16:44.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:16:45.020 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T22:16:45.020 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:16:45.020 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:16:45.021 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:16:45.021 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:45.021 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:45.021 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:16:45.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T22:16:45.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T22:16:45.121 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:16:45.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T22:16:45.136 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:16:42.346+0000 7f3d73905e00 0 load: jerasure load: lrc 2024-09-15T22:16:45.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2024-09-15T22:16:45.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:410: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2024-09-15T22:16:45.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: local objname=myobject 2024-09-15T22:16:45.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: setup_osds 7 2024-09-15T22:16:45.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-15T22:16:45.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T22:16:45.138 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-15T22:16:45.139 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:16:45.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:16:45.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:16:45.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:16:45.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:16:45.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:16:45.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:16:45.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:16:45.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:16:45.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:16:45.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:16:45.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:16:45.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:16:45.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T22:16:45.148 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:16:45.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f0166e9f-e0c5-4293-bbc8-47938e7629e7 2024-09-15T22:16:45.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f0166e9f-e0c5-4293-bbc8-47938e7629e7' 2024-09-15T22:16:45.148 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 f0166e9f-e0c5-4293-bbc8-47938e7629e7 2024-09-15T22:16:45.148 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:16:45.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDNXOdmcVcQChAAX1imxHOlk7wSZ9LScEynqw== 2024-09-15T22:16:45.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDNXOdmcVcQChAAX1imxHOlk7wSZ9LScEynqw=="}' 2024-09-15T22:16:45.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f0166e9f-e0c5-4293-bbc8-47938e7629e7 -i td/test-erasure-eio/0/new.json 2024-09-15T22:16:45.462 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:16:45.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T22:16:45.479 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNXOdmcVcQChAAX1imxHOlk7wSZ9LScEynqw== --osd-uuid f0166e9f-e0c5-4293-bbc8-47938e7629e7 2024-09-15T22:16:45.509 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:45.506+0000 7f6d91b5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:45.530 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:45.526+0000 7f6d91b5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:45.531 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:45.530+0000 7f6d91b5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:45.531 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:45.530+0000 7f6d91b5f7c0 -1 bdev(0x563c00e7bc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:16:45.531 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:45.530+0000 7f6d91b5f7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T22:16:51.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T22:16:51.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:16:51.241 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T22:16:51.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:16:51.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:16:51.814 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T22:16:51.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:16:51.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:16:51.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:16:51.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:16:51.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:16:51.846 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:51.842+0000 7f560d1237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:51.860 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:51.858+0000 7f560d1237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:51.861 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:51.858+0000 7f560d1237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:52.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:16:52.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:16:52.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:16:52.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:16:52.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:16:52.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:16:52.281 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:16:52.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:16:52.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:52.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:16:52.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:52.897 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:52.894+0000 7f560d1237c0 -1 Falling back to public interface 2024-09-15T22:16:53.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:53.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:16:53.759 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:16:53.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:16:53.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:53.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:16:54.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:55.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:55.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:16:55.228 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:16:55.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:16:55.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:55.229 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:16:55.549 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:16:55.546+0000 7f560d1237c0 -1 osd.0 0 log_to_monitors true 2024-09-15T22:16:55.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:56.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:56.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:16:56.659 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:16:56.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:16:56.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:56.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:16:57.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:58.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:58.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:16:58.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:16:58.115 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:16:58.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:58.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:16:58.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:59.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:59.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:16:59.749 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:16:59.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:16:59.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:59.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:00.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:01.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:01.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:01.212 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:17:01.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:17:01.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:01.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:01.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:02.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:02.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:02.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:17:02.700 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:17:02.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:02.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:03.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:04.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:04.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:04.175 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:17:04.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:17:04.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:04.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:04.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:05.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:05.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:05.704 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:17:05.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:17:05.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:05.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:06.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:07.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:07.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:07.164 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:17:07.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:17:07.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:07.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:07.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:08.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:08.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:08.728 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:17:08.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:17:08.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:08.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:09.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:10.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:10.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:10.295 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:17:10.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:17:10.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:10.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:11.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:12.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:12.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:12.389 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:17:12.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:17:12.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:12.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:13.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:14.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:14.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:14.245 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:17:14.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:17:14.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:14.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:15.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:16.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:16.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:16.138 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:17:16.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:17:16.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:16.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:16.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:17.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:17.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:17.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:17:17.908 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:17:17.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:17.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:18.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:19.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:19.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:19.711 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:17:19.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:17:19.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:19.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:17:20.314 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1240057757,v1:127.0.0.1:6803/1240057757] [v2:127.0.0.1:6804/1240057757,v1:127.0.0.1:6805/1240057757] exists,up f0166e9f-e0c5-4293-bbc8-47938e7629e7 2024-09-15T22:17:20.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:17:20.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:17:20.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:17:20.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:17:20.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:17:20.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:17:20.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:17:20.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:17:20.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:17:20.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:17:20.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:17:20.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:17:20.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:17:20.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:17:20.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:17:20.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:17:20.322 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:17:20.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6117203e-3a60-4639-855d-0eaf4e286329 2024-09-15T22:17:20.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6117203e-3a60-4639-855d-0eaf4e286329' 2024-09-15T22:17:20.322 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 6117203e-3a60-4639-855d-0eaf4e286329 2024-09-15T22:17:20.322 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:17:20.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDwXOdmyp8mFBAAwVd/FY/bxk9mSpKCuNH1UQ== 2024-09-15T22:17:20.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDwXOdmyp8mFBAAwVd/FY/bxk9mSpKCuNH1UQ=="}' 2024-09-15T22:17:20.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6117203e-3a60-4639-855d-0eaf4e286329 -i td/test-erasure-eio/1/new.json 2024-09-15T22:17:20.801 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:17:20.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T22:17:20.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDwXOdmyp8mFBAAwVd/FY/bxk9mSpKCuNH1UQ== --osd-uuid 6117203e-3a60-4639-855d-0eaf4e286329 2024-09-15T22:17:20.846 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:20.842+0000 7f94d019b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:20.866 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:20.866+0000 7f94d019b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:20.868 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:20.866+0000 7f94d019b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:20.868 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:20.866+0000 7f94d019b7c0 -1 bdev(0x559b1cf07c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:17:20.868 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:20.866+0000 7f94d019b7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T22:17:27.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T22:17:27.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:17:27.085 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T22:17:27.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:17:27.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:17:27.652 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T22:17:27.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:17:27.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:17:27.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:17:27.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:17:27.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:17:27.685 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:27.682+0000 7f5d4b5887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:27.703 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:27.702+0000 7f5d4b5887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:27.704 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:27.702+0000 7f5d4b5887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:28.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:17:28.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:17:28.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:17:28.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:17:28.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:17:28.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:28.135 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:17:28.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:17:28.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:28.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:28.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:28.982 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:28.978+0000 7f5d4b5887c0 -1 Falling back to public interface 2024-09-15T22:17:29.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:29.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:29.588 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:17:29.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:17:29.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:29.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:30.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:31.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:31.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:31.005 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:17:31.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:17:31.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:31.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:31.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:31.792 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:31.790+0000 7f5d4b5887c0 -1 osd.1 0 log_to_monitors true 2024-09-15T22:17:32.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:32.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:32.455 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:17:32.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:17:32.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:32.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:32.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:33.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:33.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:33.973 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:17:33.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:17:33.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:33.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:34.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:35.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:35.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:35.434 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:17:35.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:17:35.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:35.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:37.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:37.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:37.014 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:17:37.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:17:37.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:37.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:37.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:38.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:38.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:38.455 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:17:38.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:17:38.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:38.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:39.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:40.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:40.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:40.072 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:17:40.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:17:40.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:40.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:40.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:41.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:41.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:41.711 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:17:41.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:17:41.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:41.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:42.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:43.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:43.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:43.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:17:43.187 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:17:43.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:43.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:43.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:44.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:44.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:44.740 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:17:44.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:17:44.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:44.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:45.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:46.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:46.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:46.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:17:46.491 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:17:46.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:46.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:47.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:48.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:48.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:48.344 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:17:48.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:17:48.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:48.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:49.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:50.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:50.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:50.205 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:17:50.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:17:50.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:50.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:51.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:52.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:52.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:52.036 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:17:52.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:17:52.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:52.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:52.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:53.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:53.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:53.835 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:17:53.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:17:53.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:53.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:54.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:55.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:55.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:55.592 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:17:55.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:17:55.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:55.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:56.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:57.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:57.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:17:57.187 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:17:57.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:17:57.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:57.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:57.618 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2572815907,v1:127.0.0.1:6811/2572815907] [v2:127.0.0.1:6812/2572815907,v1:127.0.0.1:6813/2572815907] exists,up 6117203e-3a60-4639-855d-0eaf4e286329 2024-09-15T22:17:57.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:17:57.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:17:57.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:17:57.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:17:57.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T22:17:57.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:17:57.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:17:57.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:17:57.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:17:57.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:17:57.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:17:57.622 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:17:57.623 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:17:57.623 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:17:57.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:17:57.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:17:57.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:17:57.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:17:57.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:17:57.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:17:57.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:17:57.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:17:57.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:17:57.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:17:57.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:17:57.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:17:57.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:17:57.626 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:17:57.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=453da846-2b80-46a9-83ed-75fcd2b31ba2 2024-09-15T22:17:57.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 453da846-2b80-46a9-83ed-75fcd2b31ba2' 2024-09-15T22:17:57.626 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 453da846-2b80-46a9-83ed-75fcd2b31ba2 2024-09-15T22:17:57.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:17:57.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAVXedmXTeCJhAAN4+xyUUxjsYXFcwwqYT4Vg== 2024-09-15T22:17:57.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAVXedmXTeCJhAAN4+xyUUxjsYXFcwwqYT4Vg=="}' 2024-09-15T22:17:57.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 453da846-2b80-46a9-83ed-75fcd2b31ba2 -i td/test-erasure-eio/2/new.json 2024-09-15T22:17:58.205 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:17:58.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T22:17:58.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVXedmXTeCJhAAN4+xyUUxjsYXFcwwqYT4Vg== --osd-uuid 453da846-2b80-46a9-83ed-75fcd2b31ba2 2024-09-15T22:17:58.254 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:58.249+0000 7fb349a107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:58.276 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:58.273+0000 7fb349a107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:58.278 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:58.273+0000 7fb349a107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:58.278 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:58.273+0000 7fb349a107c0 -1 bdev(0x55ab689bbc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:17:58.278 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:17:58.277+0000 7fb349a107c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T22:18:04.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T22:18:04.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:18:04.977 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T22:18:04.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:18:04.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:18:05.556 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:18:05.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:18:05.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:18:05.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:18:05.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:18:05.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:18:05.590 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:05.585+0000 7f60692e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:18:05.607 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:05.605+0000 7f60692e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:18:05.608 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:05.605+0000 7f60692e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:18:05.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:18:05.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:18:05.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:18:05.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:18:05.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:18:05.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:05.983 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:18:05.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:18:05.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:05.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:06.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:07.102 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:07.097+0000 7f60692e47c0 -1 Falling back to public interface 2024-09-15T22:18:07.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:07.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:07.436 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:18:07.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:18:07.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:07.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:07.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:08.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:08.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:08.862 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:18:08.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:18:08.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:08.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:09.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:09.916 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:09.913+0000 7f60692e47c0 -1 osd.2 0 log_to_monitors true 2024-09-15T22:18:10.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:10.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:10.300 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:18:10.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:18:10.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:10.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:11.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:12.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:12.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:12.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:18:12.023 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:18:12.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:12.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:12.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:13.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:13.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:13.470 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:18:13.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:18:13.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:13.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:14.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:15.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:15.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:15.033 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:18:15.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:18:15.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:15.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:15.574 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:16.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:16.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:16.576 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:18:16.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:18:16.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:16.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:17.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:18.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:18.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:18.010 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:18:18.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:18:18.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:18.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:18.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:19.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:19.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:19.493 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:18:19.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:18:19.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:19.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:20.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:21.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:21.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:21.005 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:18:21.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:18:21.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:21.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:21.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:22.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:22.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:22.491 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:18:22.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:18:22.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:22.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:22.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:23.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:23.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:23.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:18:23.977 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:18:23.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:23.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:24.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:25.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:25.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:25.529 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:18:25.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:18:25.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:25.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:26.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:27.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:27.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:27.384 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:18:27.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:18:27.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:27.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:28.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:29.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:29.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:29.119 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:18:29.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:18:29.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:29.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:29.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:30.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:30.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:30.931 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:18:30.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:18:30.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:30.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:31.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:32.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:32.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:32.516 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:18:32.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:18:32.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:32.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:33.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:34.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:34.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:34.404 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:18:34.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:18:34.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:34.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:18:34.878 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3822178326,v1:127.0.0.1:6819/3822178326] [v2:127.0.0.1:6820/3822178326,v1:127.0.0.1:6821/3822178326] exists,up 453da846-2b80-46a9-83ed-75fcd2b31ba2 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:18:34.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:18:34.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:18:34.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:18:34.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:18:34.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:18:34.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:18:34.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:18:34.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:18:34.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:18:34.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:18:34.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:18:34.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:18:34.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:18:34.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:18:34.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:18:34.885 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:18:34.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5c8252e4-9235-4d00-ab83-442278a6885a 2024-09-15T22:18:34.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 5c8252e4-9235-4d00-ab83-442278a6885a' 2024-09-15T22:18:34.885 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 5c8252e4-9235-4d00-ab83-442278a6885a 2024-09-15T22:18:34.885 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:18:34.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA6Xedmb276NRAABO41L5yIf4k8x5i9SrJ09Q== 2024-09-15T22:18:34.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA6Xedmb276NRAABO41L5yIf4k8x5i9SrJ09Q=="}' 2024-09-15T22:18:34.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5c8252e4-9235-4d00-ab83-442278a6885a -i td/test-erasure-eio/3/new.json 2024-09-15T22:18:35.398 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:18:35.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T22:18:35.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6Xedmb276NRAABO41L5yIf4k8x5i9SrJ09Q== --osd-uuid 5c8252e4-9235-4d00-ab83-442278a6885a 2024-09-15T22:18:35.444 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:35.441+0000 7fa7458527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:18:35.454 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:35.453+0000 7fa7458527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:18:35.456 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:35.453+0000 7fa7458527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:18:35.456 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:35.453+0000 7fa7458527c0 -1 bdev(0x55e371e7dc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T22:18:35.456 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:35.453+0000 7fa7458527c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T22:18:42.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T22:18:42.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:18:42.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T22:18:42.067 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T22:18:42.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:18:42.670 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T22:18:42.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T22:18:42.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:18:42.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:18:42.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:18:42.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:18:42.702 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:42.697+0000 7fa1a59707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:18:42.704 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:42.701+0000 7fa1a59707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:18:42.706 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:42.701+0000 7fa1a59707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:18:43.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T22:18:43.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:18:43.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:18:43.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:18:43.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:18:43.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:43.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:18:43.135 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:18:43.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:43.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:18:43.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:44.206 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:44.201+0000 7fa1a59707c0 -1 Falling back to public interface 2024-09-15T22:18:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:44.557 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:18:44.557 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:18:44.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:44.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:18:45.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:46.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:46.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:46.026 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:18:46.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:18:46.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:46.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:18:46.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:47.179 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:18:47.177+0000 7fa1a59707c0 -1 osd.3 0 log_to_monitors true 2024-09-15T22:18:47.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:47.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:47.546 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:18:47.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:18:47.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:47.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:18:48.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:49.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:49.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:49.067 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:18:49.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:18:49.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:49.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:18:49.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:50.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:50.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:50.856 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:18:50.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:18:50.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:50.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:18:51.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:52.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:52.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:52.343 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:18:52.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:18:52.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:52.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:18:52.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:53.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:53.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:53.805 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:18:53.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:18:53.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:53.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:18:54.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:55.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:55.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:55.379 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:18:55.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:18:55.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:55.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:18:55.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:56.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:56.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:56.826 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:18:56.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:18:56.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:56.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:18:57.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:58.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:58.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:58.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:18:58.369 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:18:58.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:58.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:18:58.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:18:59.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:18:59.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:18:59.808 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:18:59.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:18:59.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:18:59.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:19:00.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:01.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:01.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:01.526 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:19:01.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:19:01.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:01.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:19:02.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:03.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:03.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:03.393 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:19:03.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:19:03.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:03.394 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:19:04.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:05.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:05.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:05.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:19:05.226 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:19:05.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:05.227 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:19:06.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:07.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:07.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:07.295 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:19:07.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:19:07.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:07.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:19:07.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:08.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:08.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:08.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:19:08.824 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:19:08.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:08.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:19:09.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:10.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:10.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:10.893 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:19:10.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:19:10.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:10.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:19:11.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:12.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:12.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:12.392 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:19:12.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:19:12.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:12.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:19:12.940 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2555719858,v1:127.0.0.1:6827/2555719858] [v2:127.0.0.1:6828/2555719858,v1:127.0.0.1:6829/2555719858] exists,up 5c8252e4-9235-4d00-ab83-442278a6885a 2024-09-15T22:19:12.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:19:12.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:19:12.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T22:19:12.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T22:19:12.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:19:12.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:19:12.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:19:12.944 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:19:12.944 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:19:12.944 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:19:12.944 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:19:12.944 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:19:12.944 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:19:12.946 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:19:12.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:19:12.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:19:12.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:19:12.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:19:12.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=116a4916-05e7-40a4-8ba9-a93f4d5c2d91 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 116a4916-05e7-40a4-8ba9-a93f4d5c2d91' 2024-09-15T22:19:12.947 INFO:tasks.workunit.client.0.smithi104.stdout:add osd4 116a4916-05e7-40a4-8ba9-a93f4d5c2d91 2024-09-15T22:19:12.948 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:19:12.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBgXedmuOrdORAAwa+aFD/u/wQBGyvzZ9gECA== 2024-09-15T22:19:12.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBgXedmuOrdORAAwa+aFD/u/wQBGyvzZ9gECA=="}' 2024-09-15T22:19:12.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 116a4916-05e7-40a4-8ba9-a93f4d5c2d91 -i td/test-erasure-eio/4/new.json 2024-09-15T22:19:13.828 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:19:13.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T22:19:13.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBgXedmuOrdORAAwa+aFD/u/wQBGyvzZ9gECA== --osd-uuid 116a4916-05e7-40a4-8ba9-a93f4d5c2d91 2024-09-15T22:19:13.884 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:13.880+0000 7f8a050477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:19:13.937 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:13.932+0000 7f8a050477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:19:13.938 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:13.932+0000 7f8a050477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:19:13.938 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:13.932+0000 7f8a050477c0 -1 bdev(0x563f4ca2fc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-15T22:19:13.938 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:13.936+0000 7f8a050477c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T22:19:20.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T22:19:20.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:19:20.166 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd4 key to auth repository 2024-09-15T22:19:20.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-15T22:19:20.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:19:20.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-15T22:19:20.744 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.4 2024-09-15T22:19:20.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:19:20.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:19:20.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:19:20.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:19:20.775 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:20.772+0000 7f6bf3e767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:19:20.777 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:20.772+0000 7f6bf3e767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:19:20.779 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:20.776+0000 7f6bf3e767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:19:21.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-15T22:19:21.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:19:21.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-15T22:19:21.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:19:21.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:19:21.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:21.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:19:21.182 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:19:21.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:21.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:21.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:22.050 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:22.044+0000 7f6bf3e767c0 -1 Falling back to public interface 2024-09-15T22:19:22.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:22.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:22.620 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:19:22.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:19:22.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:22.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:23.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:24.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:24.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:24.104 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:19:24.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:19:24.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:24.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:24.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:24.970 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:24.964+0000 7f6bf3e767c0 -1 osd.4 0 log_to_monitors true 2024-09-15T22:19:25.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:25.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:25.608 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:19:25.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:19:25.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:25.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:26.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:27.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:27.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:27.225 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:19:27.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:19:27.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:27.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:27.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:28.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:28.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:28.941 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:19:28.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:19:28.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:28.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:29.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:30.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:30.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:30.403 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:19:30.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:19:30.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:30.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:30.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:31.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:31.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:31.913 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:19:31.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:19:31.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:31.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:32.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:33.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:33.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:33.353 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:19:33.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:19:33.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:33.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:33.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:34.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:34.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:34.837 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:19:34.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:19:34.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:34.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:35.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:36.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:36.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:36.284 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:19:36.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:19:36.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:36.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:36.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:37.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:37.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:37.840 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:19:37.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:19:37.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:37.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:38.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:39.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:39.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:39.294 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:19:39.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:19:39.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:39.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:40.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:41.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:41.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:41.049 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:19:41.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:19:41.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:41.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:41.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:42.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:42.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:42.842 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:19:42.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:19:42.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:42.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:43.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:44.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:44.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:44.510 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:19:44.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:19:44.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:44.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:45.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:46.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:46.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:46.893 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:19:46.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:19:46.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:46.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:47.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:48.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:48.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:48.938 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:19:48.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:19:48.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:48.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:49.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:19:50.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:19:50.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:19:50.749 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:19:50.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:19:50.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:19:50.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T22:19:51.546 INFO:tasks.workunit.client.0.smithi104.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1329274103,v1:127.0.0.1:6835/1329274103] [v2:127.0.0.1:6836/1329274103,v1:127.0.0.1:6837/1329274103] exists,up 116a4916-05e7-40a4-8ba9-a93f4d5c2d91 2024-09-15T22:19:51.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:19:51.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:19:51.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:19:51.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:19:51.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-15T22:19:51.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:19:51.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:19:51.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-15T22:19:51.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:19:51.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-15T22:19:51.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:19:51.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:19:51.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:19:51.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:19:51.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-15T22:19:51.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-15T22:19:51.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:19:51.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:19:51.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:19:51.552 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:19:51.552 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:19:51.552 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:19:51.552 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:19:51.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:19:51.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:19:51.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:19:51.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-15T22:19:51.554 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:19:51.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=22ef5a8a-de03-4866-a22c-1a442fa0fbbd 2024-09-15T22:19:51.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 22ef5a8a-de03-4866-a22c-1a442fa0fbbd' 2024-09-15T22:19:51.555 INFO:tasks.workunit.client.0.smithi104.stdout:add osd5 22ef5a8a-de03-4866-a22c-1a442fa0fbbd 2024-09-15T22:19:51.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:19:51.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCHXedmEUEkIhAATqqDkIsYqr3v9lHMHiyeEQ== 2024-09-15T22:19:51.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCHXedmEUEkIhAATqqDkIsYqr3v9lHMHiyeEQ=="}' 2024-09-15T22:19:51.573 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 22ef5a8a-de03-4866-a22c-1a442fa0fbbd -i td/test-erasure-eio/5/new.json 2024-09-15T22:19:52.259 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:19:52.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-15T22:19:52.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHXedmEUEkIhAATqqDkIsYqr3v9lHMHiyeEQ== --osd-uuid 22ef5a8a-de03-4866-a22c-1a442fa0fbbd 2024-09-15T22:19:52.313 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:52.308+0000 7f42ea74d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:19:52.315 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:52.312+0000 7f42ea74d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:19:52.316 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:52.312+0000 7f42ea74d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:19:52.317 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:52.312+0000 7f42ea74d7c0 -1 bdev(0x55d23d59dc00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-15T22:19:52.317 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:52.312+0000 7f42ea74d7c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-15T22:19:59.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-15T22:19:59.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:19:59.112 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd5 key to auth repository 2024-09-15T22:19:59.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-15T22:19:59.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:19:59.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-15T22:19:59.687 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.5 2024-09-15T22:19:59.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:19:59.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:19:59.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:19:59.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:19:59.719 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:59.720+0000 7fb9d62ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:19:59.736 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:59.736+0000 7fb9d62ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:19:59.737 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:19:59.736+0000 7fb9d62ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:20:00.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-15T22:20:00.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:20:00.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T22:20:00.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:20:00.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:20:00.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:00.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:20:00.166 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:20:00.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:00.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:00.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:00.838 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:00.836+0000 7fb9d62ed7c0 -1 Falling back to public interface 2024-09-15T22:20:01.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:01.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:01.623 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:20:01.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:20:01.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:01.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:02.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:03.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:03.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:03.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:20:03.063 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:20:03.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:03.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:03.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:03.552 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:03.552+0000 7fb9d62ed7c0 -1 osd.5 0 log_to_monitors true 2024-09-15T22:20:04.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:04.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:04.485 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:20:04.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:20:04.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:04.486 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:05.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:06.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:06.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:06.076 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:20:06.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:20:06.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:06.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:06.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:07.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:07.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:07.978 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:20:07.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:20:07.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:07.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:08.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:09.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:09.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:09.469 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:20:09.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:20:09.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:09.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:09.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:10.935 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:10.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:10.936 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:20:10.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:20:10.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:10.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:11.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:12.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:12.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:12.508 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:20:12.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:20:12.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:12.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:13.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:14.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:14.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:14.109 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:20:14.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:20:14.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:14.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:14.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:15.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:15.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:15.545 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:20:15.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:20:15.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:15.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:15.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:16.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:16.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:16.990 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:20:16.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:20:16.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:16.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:17.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:18.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:18.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:18.524 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:20:18.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:20:18.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:18.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:19.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:20.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:20.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:20.722 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:20:20.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:20:20.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:20.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:21.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:22.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:22.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:22.619 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:20:22.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:20:22.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:22.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:23.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:24.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:24.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:24.477 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:20:24.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:20:24.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:24.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:25.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:26.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:26.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:26.617 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:20:26.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:20:26.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:26.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:27.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:28.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:28.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:28.766 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:20:28.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:20:28.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:28.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T22:20:29.234 INFO:tasks.workunit.client.0.smithi104.stdout:osd.5 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/982965220,v1:127.0.0.1:6843/982965220] [v2:127.0.0.1:6844/982965220,v1:127.0.0.1:6845/982965220] exists,up 22ef5a8a-de03-4866-a22c-1a442fa0fbbd 2024-09-15T22:20:29.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:20:29.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:20:29.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:20:29.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:20:29.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-15T22:20:29.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:20:29.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:20:29.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:20:29.238 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:20:29.239 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:20:29.239 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:20:29.239 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:20:29.240 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:20:29.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:20:29.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:20:29.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:20:29.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:20:29.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:20:29.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:20:29.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:20:29.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:20:29.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:20:29.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:20:29.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:20:29.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:20:29.241 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-15T22:20:29.241 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:20:29.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b84e3f16-3ee1-49bc-85e1-5421564ea343 2024-09-15T22:20:29.242 INFO:tasks.workunit.client.0.smithi104.stdout:add osd6 b84e3f16-3ee1-49bc-85e1-5421564ea343 2024-09-15T22:20:29.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 b84e3f16-3ee1-49bc-85e1-5421564ea343' 2024-09-15T22:20:29.242 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:20:29.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCtXedmoCqnDxAAD0/Gfs9phagI12Mg5MfD+g== 2024-09-15T22:20:29.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCtXedmoCqnDxAAD0/Gfs9phagI12Mg5MfD+g=="}' 2024-09-15T22:20:29.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b84e3f16-3ee1-49bc-85e1-5421564ea343 -i td/test-erasure-eio/6/new.json 2024-09-15T22:20:29.882 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:20:29.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-15T22:20:29.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtXedmoCqnDxAAD0/Gfs9phagI12Mg5MfD+g== --osd-uuid b84e3f16-3ee1-49bc-85e1-5421564ea343 2024-09-15T22:20:29.940 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:29.939+0000 7f06c070d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:20:30.035 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:30.031+0000 7f06c070d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:20:30.037 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:30.035+0000 7f06c070d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:20:30.037 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:30.035+0000 7f06c070d7c0 -1 bdev(0x561ef6b97c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-15T22:20:30.037 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:30.035+0000 7f06c070d7c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-15T22:20:38.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-15T22:20:38.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:20:38.050 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd6 key to auth repository 2024-09-15T22:20:38.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-15T22:20:38.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:20:38.658 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.6 2024-09-15T22:20:38.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-15T22:20:38.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:20:38.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:20:38.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:20:38.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:20:38.690 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:38.687+0000 7f3d802207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:20:38.698 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:38.695+0000 7f3d802207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:20:38.699 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:38.699+0000 7f3d802207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:20:39.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-15T22:20:39.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:20:39.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-15T22:20:39.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:20:39.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:20:39.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:39.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:20:39.095 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:20:39.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:39.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:39.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:40.014 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:40.011+0000 7f3d802207c0 -1 Falling back to public interface 2024-09-15T22:20:40.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:40.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:40.509 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:20:40.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:20:40.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:40.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:40.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:41.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:41.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:41.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:20:41.930 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:20:41.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:41.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:42.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:43.071 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:20:43.071+0000 7f3d802207c0 -1 osd.6 0 log_to_monitors true 2024-09-15T22:20:43.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:43.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:43.358 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:20:43.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:20:43.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:43.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:43.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:44.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:44.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:44.902 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:20:44.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:20:44.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:44.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:45.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:46.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:46.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:46.492 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:20:46.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:20:46.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:46.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:47.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:48.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:48.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:48.299 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:20:48.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:20:48.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:48.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:48.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:49.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:49.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:49.867 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:20:49.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:20:49.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:49.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:50.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:51.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:51.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:51.325 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:20:51.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:20:51.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:51.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:51.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:52.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:52.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:52.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:20:52.891 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:20:52.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:52.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:53.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:54.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:54.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:54.399 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:20:54.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:20:54.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:54.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:54.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:55.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:55.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:55.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:20:55.936 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:20:55.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:55.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:56.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:57.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:57.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:57.357 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:20:57.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:20:57.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:57.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:20:58.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:20:59.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:20:59.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:20:59.314 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:20:59.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:20:59.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:20:59.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:21:00.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:01.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:01.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:01.111 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:21:01.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:21:01.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:01.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:21:01.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:02.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:02.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:02.970 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:21:02.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:21:02.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:02.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:21:04.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:05.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:05.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:05.037 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:21:05.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:21:05.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:05.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:21:05.935 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:06.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:06.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:06.938 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:21:06.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:21:06.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:06.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:21:07.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:08.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:08.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:08.747 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:21:08.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:21:08.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:08.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T22:21:09.275 INFO:tasks.workunit.client.0.smithi104.stdout:osd.6 up in weight 1 up_from 59 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3202599885,v1:127.0.0.1:6851/3202599885] [v2:127.0.0.1:6852/3202599885,v1:127.0.0.1:6853/3202599885] exists,up b84e3f16-3ee1-49bc-85e1-5421564ea343 2024-09-15T22:21:09.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:21:09.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:21:09.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:21:09.277 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:21:09.277 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:21:09.277 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:21:09.277 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:21:09.277 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:09.278 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:09.278 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:21:09.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:21:09.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:21:09.389 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:21:09.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:21:09.408 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:16:52.906+0000 7f560d1237c0 0 load: jerasure load: lrc 2024-09-15T22:21:09.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:415: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2024-09-15T22:21:09.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:416: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T22:21:09.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:21:09.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:21:09.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T22:21:09.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:21:09.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T22:21:09.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:21:09.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T22:21:10.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:21:10.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:21:11.389 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:21:11.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:21:12.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:21:12.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:21:12.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:21:12.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:21:12.408 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:21:12.409 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:21:12.409 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:21:12.410 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:21:12.410 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:21:12.410 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:21:12.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:21:12.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:21:12.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:21:12.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:21:12.537 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:21:12.537 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:21:13.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:21:13.268 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:21:13.268 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:21:13.268 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:21:13.268 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:21:13.268 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:21:13.268 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T22:21:13.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:21:13.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:13.269 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:21:13.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803827 2024-09-15T22:21:13.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803827 2024-09-15T22:21:13.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827' 2024-09-15T22:21:13.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:13.428 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:21:13.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574892 2024-09-15T22:21:13.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574892 2024-09-15T22:21:13.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574892' 2024-09-15T22:21:13.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:13.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:21:13.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378663 2024-09-15T22:21:13.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378663 2024-09-15T22:21:13.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574892 2-81604378663' 2024-09-15T22:21:13.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:13.719 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:21:13.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117022 2024-09-15T22:21:13.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117022 2024-09-15T22:21:13.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574892 2-81604378663 3-115964117022' 2024-09-15T22:21:13.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:13.867 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T22:21:14.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757273 2024-09-15T22:21:14.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757273 2024-09-15T22:21:14.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574892 2-81604378663 3-115964117022 4-163208757273' 2024-09-15T22:21:14.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:14.021 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T22:21:14.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397524 2024-09-15T22:21:14.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397524 2024-09-15T22:21:14.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574892 2-81604378663 3-115964117022 4-163208757273 5-210453397524' 2024-09-15T22:21:14.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:14.170 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T22:21:14.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=253403070472 2024-09-15T22:21:14.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 253403070472 2024-09-15T22:21:14.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574892 2-81604378663 3-115964117022 4-163208757273 5-210453397524 6-253403070472' 2024-09-15T22:21:14.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:14.327 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803827 2024-09-15T22:21:14.327 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:14.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:21:14.329 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803827 2024-09-15T22:21:14.329 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:14.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803827 2024-09-15T22:21:14.330 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803827 2024-09-15T22:21:14.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-09-15T22:21:14.330 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:21:14.742 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803827 -lt 25769803827 2024-09-15T22:21:14.742 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:14.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574892 2024-09-15T22:21:14.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:14.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:21:14.745 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574892 2024-09-15T22:21:14.745 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:14.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574892 2024-09-15T22:21:14.747 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574892 2024-09-15T22:21:14.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574892' 2024-09-15T22:21:14.747 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:21:15.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574892 -lt 55834574892 2024-09-15T22:21:15.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:15.260 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:15.260 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378663 2024-09-15T22:21:15.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:21:15.262 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:15.262 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378663 2024-09-15T22:21:15.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378663 2024-09-15T22:21:15.264 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 81604378663 2024-09-15T22:21:15.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378663' 2024-09-15T22:21:15.264 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:21:15.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378663 -lt 81604378663 2024-09-15T22:21:15.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:15.783 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117022 2024-09-15T22:21:15.783 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:15.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:21:15.785 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117022 2024-09-15T22:21:15.785 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:15.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117022 2024-09-15T22:21:15.786 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 115964117022 2024-09-15T22:21:15.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117022' 2024-09-15T22:21:15.787 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:21:16.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117022 -lt 115964117022 2024-09-15T22:21:16.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:16.189 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757273 2024-09-15T22:21:16.189 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:16.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T22:21:16.191 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757273 2024-09-15T22:21:16.191 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:16.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757273 2024-09-15T22:21:16.193 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 163208757273 2024-09-15T22:21:16.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757273' 2024-09-15T22:21:16.193 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T22:21:16.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757274 -lt 163208757273 2024-09-15T22:21:16.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:16.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-210453397524 2024-09-15T22:21:16.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:16.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T22:21:16.628 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-210453397524 2024-09-15T22:21:16.629 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:16.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397524 2024-09-15T22:21:16.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 210453397524' 2024-09-15T22:21:16.630 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 210453397524 2024-09-15T22:21:16.630 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T22:21:17.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397524 -lt 210453397524 2024-09-15T22:21:17.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:17.091 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-253403070472 2024-09-15T22:21:17.091 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:17.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T22:21:17.094 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-253403070472 2024-09-15T22:21:17.094 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:17.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=253403070472 2024-09-15T22:21:17.095 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.6 seq 253403070472 2024-09-15T22:21:17.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 253403070472' 2024-09-15T22:21:17.096 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T22:21:17.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 253403070472 -lt 253403070472 2024-09-15T22:21:17.527 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:21:17.527 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:21:17.528 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:21:18.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:21:18.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:21:18.068 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:21:18.068 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:21:18.068 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:21:18.068 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:21:18.068 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:21:18.068 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:21:18.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:21:18.483 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:21:18.484 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:21:18.484 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:21:19.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:21:19.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:21:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-15T22:21:19.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T22:21:19.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:21:19.316 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-15T22:21:19.316 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:21:19.316 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-15T22:21:19.317 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T22:21:19.317 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:21:19.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:21:19.737 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:21:19.737 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:21:19.737 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:21:19.737 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T22:21:19.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2024-09-15T22:21:19.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-09-15T22:21:19.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:21:19.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-15T22:21:19.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:21:19.738 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T22:21:19.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:21:19.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T22:21:19.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:21:19.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-15T22:21:19.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:21:19.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:21:19.740 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:21:19.740 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-15T22:21:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:21:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:21:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:21:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:21:19.741 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:21:19.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:21:19.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:21:19.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:21:19.857 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-15T22:21:19.857 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:21:19.857 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:21:19.858 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:21:19.858 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:19.858 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:19.858 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:21:19.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:21:19.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-15T22:21:19.979 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2024-09-15T22:21:19.979 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:21:19.979 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-15T22:21:19.980 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T22:21:19.980 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:21:20.407 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:21:20.407 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:21:20.408 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:21:20.408 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:21:20.408 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T22:21:20.408 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2024-09-15T22:21:20.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: initial_osds=('3' '5' '1' '0' '6') 2024-09-15T22:21:20.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: local -a initial_osds 2024-09-15T22:21:20.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:422: TEST_ec_single_recovery_error: local last_osd=6 2024-09-15T22:21:20.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2024-09-15T22:21:20.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:21:20.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:21:20.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:21:20.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:21:20.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:21:21.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:21:21.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:425: TEST_ec_single_recovery_error: ceph osd down 6 2024-09-15T22:21:22.200 INFO:tasks.workunit.client.0.smithi104.stderr:osd.6 is already down. 2024-09-15T22:21:22.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_single_recovery_error: ceph osd out 6 2024-09-15T22:21:22.969 INFO:tasks.workunit.client.0.smithi104.stderr:osd.6 is already out. 2024-09-15T22:21:22.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:429: TEST_ec_single_recovery_error: wait_for_clean 2024-09-15T22:21:22.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:21:22.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:21:22.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:21:22.987 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:21:22.988 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:21:22.988 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:21:22.988 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:21:22.988 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:21:22.988 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:21:23.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:21:23.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:21:23.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:21:23.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:21:23.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:21:23.092 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:21:23.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:21:23.880 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:21:23.880 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:21:23.880 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:21:23.880 INFO:tasks.workunit.client.0.smithi104.stderr:4 2024-09-15T22:21:23.880 INFO:tasks.workunit.client.0.smithi104.stderr:5 2024-09-15T22:21:23.880 INFO:tasks.workunit.client.0.smithi104.stderr:6' 2024-09-15T22:21:23.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:21:23.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:23.881 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:21:24.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803831 2024-09-15T22:21:24.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803831 2024-09-15T22:21:24.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831' 2024-09-15T22:21:24.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:24.068 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:21:24.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574896 2024-09-15T22:21:24.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574896 2024-09-15T22:21:24.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574896' 2024-09-15T22:21:24.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:24.314 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:21:24.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378667 2024-09-15T22:21:24.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378667 2024-09-15T22:21:24.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574896 2-81604378667' 2024-09-15T22:21:24.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:24.472 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:21:24.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117026 2024-09-15T22:21:24.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117026 2024-09-15T22:21:24.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574896 2-81604378667 3-115964117026' 2024-09-15T22:21:24.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:24.624 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T22:21:24.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757277 2024-09-15T22:21:24.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757277 2024-09-15T22:21:24.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574896 2-81604378667 3-115964117026 4-163208757277' 2024-09-15T22:21:24.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:24.783 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T22:21:24.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397528 2024-09-15T22:21:24.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397528 2024-09-15T22:21:24.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574896 2-81604378667 3-115964117026 4-163208757277 5-210453397528' 2024-09-15T22:21:24.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:21:24.939 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T22:21:25.051 INFO:tasks.workunit.client.0.smithi104.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-15T22:21:25.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T22:21:25.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T22:21:25.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T22:21:25.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:25.054 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803831 2024-09-15T22:21:25.054 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:25.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:21:25.056 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803831 2024-09-15T22:21:25.056 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:25.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803831 2024-09-15T22:21:25.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-09-15T22:21:25.057 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803831 2024-09-15T22:21:25.057 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:21:25.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803831 -lt 25769803831 2024-09-15T22:21:25.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:25.635 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574896 2024-09-15T22:21:25.635 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:25.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:21:25.637 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574896 2024-09-15T22:21:25.637 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:25.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574896 2024-09-15T22:21:25.639 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574896 2024-09-15T22:21:25.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574896' 2024-09-15T22:21:25.639 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:21:26.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574896 -lt 55834574896 2024-09-15T22:21:26.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:26.131 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378667 2024-09-15T22:21:26.131 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:26.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:21:26.133 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378667 2024-09-15T22:21:26.133 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:26.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378667 2024-09-15T22:21:26.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378667' 2024-09-15T22:21:26.135 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 81604378667 2024-09-15T22:21:26.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:21:26.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378668 -lt 81604378667 2024-09-15T22:21:26.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:26.596 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117026 2024-09-15T22:21:26.596 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:26.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:21:26.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:26.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117026 2024-09-15T22:21:26.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117026 2024-09-15T22:21:26.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117026' 2024-09-15T22:21:26.600 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 115964117026 2024-09-15T22:21:26.601 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:21:27.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117026 -lt 115964117026 2024-09-15T22:21:27.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:27.037 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757277 2024-09-15T22:21:27.037 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:27.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T22:21:27.040 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757277 2024-09-15T22:21:27.040 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:27.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757277 2024-09-15T22:21:27.041 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.4 seq 163208757277 2024-09-15T22:21:27.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757277' 2024-09-15T22:21:27.042 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T22:21:27.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757278 -lt 163208757277 2024-09-15T22:21:27.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:21:27.483 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-210453397528 2024-09-15T22:21:27.483 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:21:27.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T22:21:27.485 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-210453397528 2024-09-15T22:21:27.485 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:21:27.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397528 2024-09-15T22:21:27.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 210453397528' 2024-09-15T22:21:27.487 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.5 seq 210453397528 2024-09-15T22:21:27.487 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T22:21:27.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397528 -lt 210453397528 2024-09-15T22:21:27.954 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:21:27.955 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:21:27.955 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:21:28.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:21:28.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:21:28.536 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:21:28.536 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:21:28.536 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:21:28.536 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:21:28.536 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:21:28.536 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:21:28.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T22:21:28.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:21:28.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:21:28.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:21:29.479 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T22:21:29.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T22:21:29.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T22:21:29.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T22:21:29.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T22:21:29.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T22:21:29.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T22:21:29.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:21:29.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:21:29.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:21:29.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:21:29.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:21:29.582 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:21:29.582 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:21:30.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T22:21:30.003 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:21:30.004 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:21:30.004 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:21:30.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T22:21:30.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T22:21:30.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T22:21:30.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T22:21:30.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T22:21:30.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T22:21:30.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T22:21:30.548 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T22:21:30.548 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T22:21:31.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=544 2024-09-15T22:21:31.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 544 '!=' null 2024-09-15T22:21:31.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-15T22:21:31.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T22:21:31.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T22:21:31.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T22:21:31.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:21:31.214 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:21:31.214 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:21:31.214 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:21:31.214 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:21:31.214 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:21:31.214 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:21:31.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:21:31.623 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:21:31.624 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:21:31.624 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:21:32.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:21:32.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:21:32.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:21:32.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-15T22:21:32.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:21:32.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:21:32.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-15T22:21:32.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:21:32.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:21:32.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-15T22:21:32.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:21:32.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:21:32.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:21:32.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:21:32.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:21:32.858 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:21:32.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:21:33.633 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:21:33.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:21:33.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:21:33.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:21:33.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:21:33.650 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:21:33.650 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:21:33.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:21:33.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:21:33.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:21:33.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:21:33.965 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:21:33.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:21:33.967 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:21:33.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:21:33.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:21:33.969 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:21:33.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:21:33.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:21:33.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:21:33.971 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:21:33.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:21:33.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:21:33.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:21:34.015 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:21:34.016 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:34.016 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:34.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:21:34.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:21:34.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:21:34.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T22:21:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T22:21:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T22:21:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T22:21:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:21:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:21:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:21:34.018 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:21:34.018 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:21:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:21:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:21:34.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:21:34.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:21:34.020 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:21:34.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:21:34.022 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:21:34.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:21:34.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:21:34.023 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:21:34.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:21:34.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:21:34.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:21:34.025 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:21:34.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:21:34.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:21:34.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:21:34.028 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:21:34.028 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:34.028 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:34.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:21:34.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:21:34.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:21:34.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T22:21:34.031 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:21:34.031 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:34.031 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:34.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T22:21:34.033 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:21:34.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:21:34.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:21:34.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T22:21:34.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T22:21:34.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T22:21:34.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:21:34.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:21:34.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:21:34.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T22:21:34.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T22:21:34.477 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:21:34.477 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:21:34.477 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:21:34.478 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:21:34.478 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:34.478 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:34.478 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:21:34.479 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T22:21:34.677 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:21:34.678 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:21:34.679 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:21:34.679 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:21:34.679 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:21:34.679 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:21:34.679 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:21:34.680 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:21:34.680 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:21:34.680 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:21:34.680 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:34.681 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:34.681 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:21:34.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:21:34.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T22:21:34.795 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:21:34.796 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:21:34.796 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:21:34.796 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:21:34.797 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:21:34.797 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:21:34.797 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:21:34.797 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:21:34.797 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:21:34.797 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:34.797 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:34.798 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:21:34.798 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:21:34.798 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T22:21:34.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T22:21:34.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T22:21:34.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:21:34.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:21:34.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:21:34.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T22:21:34.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:21:35.839 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:21:35.839 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:21:35.839 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:21:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:21:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:35.839 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:21:35.840 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:21:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:21:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T22:21:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T22:21:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T22:21:36.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:21:37.219 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T22:21:37.219 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:21:37.220 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:21:37.220 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:21:37.220 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:37.220 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:37.221 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:21:37.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T22:21:37.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T22:21:37.330 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:21:37.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T22:21:37.346 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:21:34.522+0000 7fdd6f8dce00 0 load: jerasure load: lrc 2024-09-15T22:21:37.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2024-09-15T22:21:37.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2024-09-15T22:21:37.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_0: setup_osds 4 2024-09-15T22:21:37.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T22:21:37.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T22:21:37.348 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T22:21:37.349 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T22:21:37.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:21:37.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:21:37.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:21:37.354 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:21:37.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:21:37.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:21:37.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:21:37.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:21:37.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:21:37.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:21:37.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T22:21:37.356 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:21:37.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1c0d753d-71e7-446e-aeca-c871e77e00ab 2024-09-15T22:21:37.357 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 1c0d753d-71e7-446e-aeca-c871e77e00ab 2024-09-15T22:21:37.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1c0d753d-71e7-446e-aeca-c871e77e00ab' 2024-09-15T22:21:37.357 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:21:37.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDxXedmmQmGFhAANGYLwS2CJXJCfwrKHgca/A== 2024-09-15T22:21:37.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDxXedmmQmGFhAANGYLwS2CJXJCfwrKHgca/A=="}' 2024-09-15T22:21:37.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1c0d753d-71e7-446e-aeca-c871e77e00ab -i td/test-erasure-eio/0/new.json 2024-09-15T22:21:37.677 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:21:37.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T22:21:37.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxXedmmQmGFhAANGYLwS2CJXJCfwrKHgca/A== --osd-uuid 1c0d753d-71e7-446e-aeca-c871e77e00ab 2024-09-15T22:21:37.723 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:21:37.718+0000 7f4b0cee67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:21:37.745 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:21:37.742+0000 7f4b0cee67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:21:37.746 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:21:37.742+0000 7f4b0cee67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:21:37.746 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:21:37.742+0000 7f4b0cee67c0 -1 bdev(0x5566dac2bc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:21:37.746 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:21:37.742+0000 7f4b0cee67c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T22:21:43.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T22:21:43.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:21:43.839 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T22:21:43.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:21:43.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:21:44.420 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T22:21:44.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:21:44.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:21:44.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:21:44.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:21:44.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:21:44.452 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:21:44.450+0000 7fdebc1c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:21:44.482 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:21:44.478+0000 7fdebc1c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:21:44.484 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:21:44.482+0000 7fdebc1c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:21:44.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:21:44.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:21:44.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:21:44.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:21:44.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:21:44.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:44.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:21:44.872 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:21:44.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:21:44.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:45.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:45.502 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:21:45.498+0000 7fdebc1c97c0 -1 Falling back to public interface 2024-09-15T22:21:46.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:46.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:46.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:21:46.364 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:21:46.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:46.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:21:46.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:47.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:47.805 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:47.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:21:47.806 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:21:47.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:47.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:21:48.130 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:21:48.126+0000 7fdebc1c97c0 -1 osd.0 0 log_to_monitors true 2024-09-15T22:21:48.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:49.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:49.224 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:49.224 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:21:49.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:21:49.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:49.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:21:49.668 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:50.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:50.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:50.669 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:21:50.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:21:50.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:50.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:21:51.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:52.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:52.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:52.172 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:21:52.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:21:52.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:52.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:21:52.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:53.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:53.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:53.632 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:21:53.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:21:53.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:53.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:21:54.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:55.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:55.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:55.066 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:21:55.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:21:55.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:55.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:21:55.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:56.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:56.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:56.511 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:21:56.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:21:56.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:56.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:21:57.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:58.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:58.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:58.040 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:21:58.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:21:58.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:58.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:21:58.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:21:59.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:21:59.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:21:59.484 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:21:59.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:21:59.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:21:59.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:21:59.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:00.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:00.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:00.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:22:00.956 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:22:00.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:00.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:22:01.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:02.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:02.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:02.856 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:22:02.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:22:02.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:02.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:22:03.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:04.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:04.774 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:04.774 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:22:04.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:22:04.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:04.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:22:05.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:06.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:06.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:06.513 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:22:06.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:22:06.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:06.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:22:06.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:07.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:07.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:07.982 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:22:07.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:22:07.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:07.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:22:09.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:10.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:10.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:10.164 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:22:10.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:22:10.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:10.164 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:22:10.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:11.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:11.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:11.989 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:22:11.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:22:11.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:11.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:22:12.466 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1176196260,v1:127.0.0.1:6803/1176196260] [v2:127.0.0.1:6804/1176196260,v1:127.0.0.1:6805/1176196260] exists,up 1c0d753d-71e7-446e-aeca-c871e77e00ab 2024-09-15T22:22:12.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:22:12.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:22:12.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:22:12.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:22:12.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T22:22:12.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:22:12.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:22:12.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:22:12.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:22:12.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:22:12.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:22:12.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:22:12.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:22:12.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:22:12.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:22:12.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:22:12.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:22:12.472 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:22:12.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b627b602-c3ed-4225-8c89-ee56604b494e 2024-09-15T22:22:12.473 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 b627b602-c3ed-4225-8c89-ee56604b494e 2024-09-15T22:22:12.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 b627b602-c3ed-4225-8c89-ee56604b494e' 2024-09-15T22:22:12.473 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:22:12.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAUXudmPmEPHRAA9yjFOVs1W2LifNG6WpwkEw== 2024-09-15T22:22:12.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAUXudmPmEPHRAA9yjFOVs1W2LifNG6WpwkEw=="}' 2024-09-15T22:22:12.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b627b602-c3ed-4225-8c89-ee56604b494e -i td/test-erasure-eio/1/new.json 2024-09-15T22:22:13.064 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:22:13.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T22:22:13.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUXudmPmEPHRAA9yjFOVs1W2LifNG6WpwkEw== --osd-uuid b627b602-c3ed-4225-8c89-ee56604b494e 2024-09-15T22:22:13.116 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:13.114+0000 7f756270a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:13.119 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:13.118+0000 7f756270a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:13.121 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:13.118+0000 7f756270a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:13.122 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:13.118+0000 7f756270a7c0 -1 bdev(0x55c63317fc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:22:13.122 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:13.118+0000 7f756270a7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T22:22:19.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T22:22:19.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:22:19.269 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T22:22:19.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:22:19.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:22:19.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:22:19.869 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T22:22:19.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:22:19.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:22:19.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:22:19.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:22:19.902 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:19.898+0000 7f53b51557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:19.904 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:19.902+0000 7f53b51557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:19.905 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:19.902+0000 7f53b51557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:20.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:22:20.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:22:20.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:22:20.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:22:20.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:22:20.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:20.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:22:20.293 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:22:20.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:20.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:20.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:21.118 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:21.114+0000 7f53b51557c0 -1 Falling back to public interface 2024-09-15T22:22:21.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:21.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:21.722 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:22:21.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:22:21.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:21.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:22.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:23.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:23.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:23.181 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:22:23.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:22:23.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:23.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:23.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:23.923 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:23.918+0000 7f53b51557c0 -1 osd.1 0 log_to_monitors true 2024-09-15T22:22:24.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:24.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:24.628 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:22:24.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:22:24.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:24.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:25.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:26.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:26.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:26.271 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:22:26.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:22:26.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:26.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:26.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:27.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:27.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:27.905 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:22:27.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:22:27.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:27.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:28.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:29.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:29.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:29.351 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:22:29.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:22:29.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:29.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:29.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:30.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:30.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:30.892 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:22:30.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:22:30.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:30.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:31.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:32.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:32.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:32.381 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:22:32.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:22:32.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:32.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:32.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:33.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:33.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:33.856 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:22:33.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:22:33.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:33.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:34.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:35.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:35.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:35.322 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:22:35.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:22:35.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:35.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:36.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:36.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:36.870 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:22:36.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:22:36.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:36.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:38.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:39.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:39.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:39.084 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:22:39.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:22:39.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:39.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:39.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:40.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:40.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:40.965 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:22:40.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:22:40.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:40.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:41.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:42.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:42.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:42.739 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:22:42.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:22:42.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:42.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:43.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:44.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:44.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:44.837 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:22:44.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:22:44.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:44.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:45.678 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:46.679 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:46.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:46.680 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:22:46.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:22:46.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:46.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:47.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:48.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:48.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:48.431 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:22:48.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:22:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:48.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:22:49.109 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4084052557,v1:127.0.0.1:6811/4084052557] [v2:127.0.0.1:6812/4084052557,v1:127.0.0.1:6813/4084052557] exists,up b627b602-c3ed-4225-8c89-ee56604b494e 2024-09-15T22:22:49.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:22:49.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:22:49.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:22:49.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:22:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:22:49.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:22:49.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:22:49.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:22:49.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:22:49.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:22:49.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:22:49.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:22:49.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:22:49.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:22:49.116 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:22:49.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=593fcfbb-be91-4680-b758-9ad4109fa61e 2024-09-15T22:22:49.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 593fcfbb-be91-4680-b758-9ad4109fa61e' 2024-09-15T22:22:49.117 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 593fcfbb-be91-4680-b758-9ad4109fa61e 2024-09-15T22:22:49.118 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:22:49.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA5Xudmt30lCBAAmyvI7UsZabH8vmU0HP8kdw== 2024-09-15T22:22:49.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA5Xudmt30lCBAAmyvI7UsZabH8vmU0HP8kdw=="}' 2024-09-15T22:22:49.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 593fcfbb-be91-4680-b758-9ad4109fa61e -i td/test-erasure-eio/2/new.json 2024-09-15T22:22:49.624 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:22:49.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T22:22:49.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5Xudmt30lCBAAmyvI7UsZabH8vmU0HP8kdw== --osd-uuid 593fcfbb-be91-4680-b758-9ad4109fa61e 2024-09-15T22:22:49.680 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:49.677+0000 7fbf825697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:49.682 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:49.677+0000 7fbf825697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:49.683 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:49.681+0000 7fbf825697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:49.684 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:49.681+0000 7fbf825697c0 -1 bdev(0x561683a3dc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:22:49.684 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:49.681+0000 7fbf825697c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T22:22:56.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T22:22:56.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:22:56.354 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T22:22:56.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:22:56.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:22:56.931 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:22:56.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:22:56.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:22:56.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:22:56.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:22:56.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:22:56.962 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:56.957+0000 7f723cca77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:56.974 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:56.969+0000 7f723cca77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:56.975 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:56.973+0000 7f723cca77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:22:57.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:22:57.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:22:57.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:22:57.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:22:57.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:22:57.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:57.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:22:57.404 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:22:57.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:57.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:22:57.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:22:58.218 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:22:58.213+0000 7f723cca77c0 -1 Falling back to public interface 2024-09-15T22:22:58.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:22:58.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:22:58.841 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:22:58.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:22:58.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:22:58.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:22:59.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:00.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:00.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:00.268 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:23:00.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:23:00.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:00.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:00.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:01.154 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:23:01.149+0000 7f723cca77c0 -1 osd.2 0 log_to_monitors true 2024-09-15T22:23:01.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:01.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:01.707 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:23:01.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:23:01.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:01.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:02.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:03.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:03.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:03.483 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:23:03.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:23:03.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:03.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:03.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:04.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:04.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:04.931 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:23:04.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:23:04.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:04.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:05.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:06.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:06.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:06.466 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:23:06.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:23:06.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:06.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:06.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:07.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:07.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:07.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:23:07.908 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:23:07.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:07.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:08.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:09.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:09.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:09.372 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:23:09.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:23:09.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:09.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:09.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:10.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:10.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:10.818 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:23:10.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:23:10.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:10.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:12.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:12.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:12.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:23:12.261 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:23:12.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:12.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:12.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:13.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:13.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:13.724 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:23:13.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:23:13.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:13.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:14.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:15.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:15.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:15.253 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:23:15.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:23:15.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:15.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:16.149 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:17.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:17.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:17.151 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:23:17.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:23:17.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:17.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:17.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:18.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:18.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:18.643 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:23:18.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:23:18.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:18.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:19.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:20.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:20.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:20.156 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:23:20.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:23:20.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:20.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:20.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:21.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:21.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:21.931 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:23:21.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:23:21.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:21.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:22.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:23.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:23.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:23.749 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:23:23.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:23:23.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:23.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:24.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:25.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:25.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:25.473 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:23:25.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:23:25.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:25.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:23:25.939 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2934746874,v1:127.0.0.1:6819/2934746874] [v2:127.0.0.1:6820/2934746874,v1:127.0.0.1:6821/2934746874] exists,up 593fcfbb-be91-4680-b758-9ad4109fa61e 2024-09-15T22:23:25.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:23:25.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:23:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:23:25.942 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:23:25.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:23:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:23:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:23:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:23:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:23:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:23:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:23:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:23:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:23:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:23:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:23:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:23:25.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:23:25.945 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:23:25.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=83950c7b-64b0-45d4-afcc-9d0c8c314315 2024-09-15T22:23:25.945 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 83950c7b-64b0-45d4-afcc-9d0c8c314315 2024-09-15T22:23:25.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 83950c7b-64b0-45d4-afcc-9d0c8c314315' 2024-09-15T22:23:25.946 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:23:25.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBdXudmw/+QORAAsVuVoZ8aYk7UNk5FCIT9Eg== 2024-09-15T22:23:25.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBdXudmw/+QORAAsVuVoZ8aYk7UNk5FCIT9Eg=="}' 2024-09-15T22:23:25.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 83950c7b-64b0-45d4-afcc-9d0c8c314315 -i td/test-erasure-eio/3/new.json 2024-09-15T22:23:27.061 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:23:27.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T22:23:27.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdXudmw/+QORAAsVuVoZ8aYk7UNk5FCIT9Eg== --osd-uuid 83950c7b-64b0-45d4-afcc-9d0c8c314315 2024-09-15T22:23:27.121 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:23:27.117+0000 7fb57f26e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:23:27.123 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:23:27.117+0000 7fb57f26e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:23:27.124 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:23:27.121+0000 7fb57f26e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:23:27.125 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:23:27.121+0000 7fb57f26e7c0 -1 bdev(0x55dad7f53c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T22:23:27.125 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:23:27.121+0000 7fb57f26e7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T22:23:33.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T22:23:33.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:23:33.915 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T22:23:33.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T22:23:33.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:23:34.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T22:23:34.487 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T22:23:34.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:23:34.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:23:34.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:23:34.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:23:34.520 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:23:34.517+0000 7fe1ac54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:23:34.543 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:23:34.537+0000 7fe1ac54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:23:34.544 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:23:34.541+0000 7fe1ac54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:23:34.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T22:23:34.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:23:34.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:23:34.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:23:34.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:23:34.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:34.920 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:23:34.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:23:34.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:34.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:35.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:23:36.077+0000 7fe1ac54f7c0 -1 Falling back to public interface 2024-09-15T22:23:36.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:36.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:36.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:23:36.351 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:23:36.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:36.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:36.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:37.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:37.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:37.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:23:37.772 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:23:37.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:37.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:38.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:39.157 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:23:39.153+0000 7fe1ac54f7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T22:23:39.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:39.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:39.254 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:23:39.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:23:39.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:39.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:39.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:40.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:40.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:40.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:23:40.899 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:23:40.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:40.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:41.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:42.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:42.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:42.511 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:23:42.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:23:42.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:42.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:43.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:44.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:44.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:44.168 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:23:44.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:23:44.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:44.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:44.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:45.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:45.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:45.612 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:23:45.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:23:45.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:45.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:46.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:47.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:47.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:47.155 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:23:47.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:23:47.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:47.156 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:47.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:48.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:48.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:48.784 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:23:48.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:23:48.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:48.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:49.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:50.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:50.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:50.510 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:23:50.510 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:23:50.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:50.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:51.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:52.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:52.056 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:23:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:23:52.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:52.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:52.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:53.492 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:53.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:53.493 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:23:53.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:23:53.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:53.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:54.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:55.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:55.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:55.757 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:23:55.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:23:55.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:55.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:56.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:57.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:57.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:57.844 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:23:57.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:23:57.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:57.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:23:58.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:23:59.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:23:59.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:23:59.629 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:23:59.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:23:59.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:23:59.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:24:00.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:24:01.567 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:24:01.567 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:01.568 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:24:01.568 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:24:01.568 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:01.568 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:24:02.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:24:03.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:24:03.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:03.819 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:24:03.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:24:03.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:03.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:24:04.464 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3457307338,v1:127.0.0.1:6827/3457307338] [v2:127.0.0.1:6828/3457307338,v1:127.0.0.1:6829/3457307338] exists,up 83950c7b-64b0-45d4-afcc-9d0c8c314315 2024-09-15T22:24:04.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:24:04.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:24:04.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:24:04.465 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:24:04.466 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:24:04.466 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:24:04.466 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:24:04.466 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:24:04.467 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:24:04.467 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:24:04.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:24:04.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:24:04.580 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:24:04.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:24:04.595 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:21:45.510+0000 7fdebc1c97c0 0 load: jerasure load: lrc 2024-09-15T22:24:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:325: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2024-09-15T22:24:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:326: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T22:24:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:24:04.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:24:04.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T22:24:04.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:24:04.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T22:24:04.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:24:04.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T22:24:05.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:24:05.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:24:06.741 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:24:06.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:24:07.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:24:07.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:24:07.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:24:07.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:24:07.762 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:24:07.763 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:24:07.763 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:24:07.763 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:24:07.763 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:24:07.763 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:24:07.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:24:07.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:24:07.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:24:07.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:24:07.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:24:07.891 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:24:08.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:24:08.342 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:24:08.343 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:24:08.343 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:24:08.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:24:08.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:08.343 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:24:08.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803806 2024-09-15T22:24:08.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803806 2024-09-15T22:24:08.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806' 2024-09-15T22:24:08.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:08.549 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:24:08.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574868 2024-09-15T22:24:08.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574868 2024-09-15T22:24:08.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574868' 2024-09-15T22:24:08.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:08.731 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:24:08.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378638 2024-09-15T22:24:08.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378638 2024-09-15T22:24:08.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574868 2-81604378638' 2024-09-15T22:24:08.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:08.936 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:24:09.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117002 2024-09-15T22:24:09.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117002 2024-09-15T22:24:09.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574868 2-81604378638 3-115964117002' 2024-09-15T22:24:09.098 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:09.099 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803806 2024-09-15T22:24:09.099 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:09.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:24:09.100 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803806 2024-09-15T22:24:09.101 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:09.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803806 2024-09-15T22:24:09.102 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803806 2024-09-15T22:24:09.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803806' 2024-09-15T22:24:09.102 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:24:09.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803806 -lt 25769803806 2024-09-15T22:24:09.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:09.576 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574868 2024-09-15T22:24:09.576 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:09.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:24:09.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574868 2024-09-15T22:24:09.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:09.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574868 2024-09-15T22:24:09.581 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574868 2024-09-15T22:24:09.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574868' 2024-09-15T22:24:09.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:24:10.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574866 -lt 55834574868 2024-09-15T22:24:10.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:24:11.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:24:11.069 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:24:11.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574868 -lt 55834574868 2024-09-15T22:24:11.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:11.496 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378638 2024-09-15T22:24:11.496 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:11.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:24:11.499 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378638 2024-09-15T22:24:11.499 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:11.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378638 2024-09-15T22:24:11.500 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 81604378638 2024-09-15T22:24:11.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378638' 2024-09-15T22:24:11.501 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:24:11.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378638 -lt 81604378638 2024-09-15T22:24:11.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:11.933 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117002 2024-09-15T22:24:11.933 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:11.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:24:11.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117002 2024-09-15T22:24:11.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:11.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117002 2024-09-15T22:24:11.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117002' 2024-09-15T22:24:11.936 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 115964117002 2024-09-15T22:24:11.937 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:24:12.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117003 -lt 115964117002 2024-09-15T22:24:12.398 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:24:12.398 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:24:12.399 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:24:12.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:24:12.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:24:12.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:24:12.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:24:12.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:24:12.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:24:12.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:24:12.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:24:13.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:24:13.371 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:24:13.371 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:24:13.372 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:24:13.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:24:13.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:24:13.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:24:13.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:328: TEST_rados_get_bad_size_shard_0: local shard_id=0 2024-09-15T22:24:13.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:329: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=10 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-81867-0-10 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81867-0-10 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:24:13.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:24:13.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81867-0-10 2024-09-15T22:24:13.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:24:13.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:24:13.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:24:13.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:24:13.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:24:13.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:24:13.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:24:13.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:24:13.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81867-0-10 td/test-erasure-eio/ORIGINAL 2024-09-15T22:24:14.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-81867-0-10 td/test-erasure-eio 0 10 set 2024-09-15T22:24:14.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-0-10 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-0-10 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:24:14.069 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-0-10 2024-09-15T22:24:14.070 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-0-10 2024-09-15T22:24:14.070 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:24:14.569 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:24:14.569 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:24:14.569 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:24:14.569 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:24:14.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:24:14.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:24:14.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2024-09-15T22:24:14.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:24:15.385 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:24:15.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-15T22:24:15.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-15T22:24:15.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-15T22:24:15.407 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records in 2024-09-15T22:24:15.408 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records out 2024-09-15T22:24:15.408 INFO:tasks.workunit.client.0.smithi104.stderr:10 bytes copied, 0.000119539 s, 83.7 kB/s 2024-09-15T22:24:15.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81867-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:15.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:24:15.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:24:15.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T22:24:15.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:24:15.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81867-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:15.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:24:15.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:24:15.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T22:24:15.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:24:15.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-15T22:24:15.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:24:15.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:24:15.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:24:15.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:24:15.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:24:15.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:24:15.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81867-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:15.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:24:15.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:24:15.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T22:24:15.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:24:15.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-15T22:24:15.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81867-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:18.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-15T22:24:18.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:24:18.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:24:18.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T22:24:18.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:24:18.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:24:18.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:24:18.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:24:18.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:24:18.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:24:18.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:24:18.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:24:18.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:24:18.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:24:18.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:24:18.168 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:24:18.168 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:24:18.168 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:24:18.169 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:24:18.169 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:24:18.169 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:24:18.169 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:24:18.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:24:18.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:24:18.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:24:18.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:24:18.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:24:18.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:24:18.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:24:18.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:24:18.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:24:18.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:24:18.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:24:18.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T22:24:18.171 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.3 2024-09-15T22:24:18.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:24:18.171 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-15T22:24:18.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T22:24:18.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:24:18.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:24:18.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:24:18.205 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:18.200+0000 7fc0497fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:24:18.207 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:18.204+0000 7fc0497fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:24:18.209 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:18.204+0000 7fc0497fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:24:18.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T22:24:18.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:24:18.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:24:18.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:24:18.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:24:18.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:18.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:24:18.643 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:24:18.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:18.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:24:19.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:24:19.454 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:19.448+0000 7fc0497fc7c0 -1 Falling back to public interface 2024-09-15T22:24:20.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:24:20.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:20.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:24:20.113 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:24:20.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:20.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:24:20.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:24:21.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:24:21.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:21.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:24:21.597 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:24:21.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:21.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:24:22.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:24:22.377 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:22.372+0000 7fc0497fc7c0 -1 osd.3 35 log_to_monitors true 2024-09-15T22:24:23.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:24:23.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:23.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:24:23.030 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:24:23.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:23.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:24:23.515 INFO:tasks.workunit.client.0.smithi104.stderr:osd.3 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [27,35) [v2:127.0.0.1:6826/1928848860,v1:127.0.0.1:6827/1928848860] [v2:127.0.0.1:6828/1928848860,v1:127.0.0.1:6829/1928848860] exists,up 83950c7b-64b0-45d4-afcc-9d0c8c314315 2024-09-15T22:24:23.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:24:23.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:24:23.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:24:23.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:24:23.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:24:23.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:24:23.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:24:23.516 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:24:23.516 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:24:23.517 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:24:23.517 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:24:23.517 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:24:23.517 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:24:23.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:24:23.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:24:23.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:24:23.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:24:23.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:24:23.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:24:24.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:24:24.152 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:24:24.152 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:24:24.152 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:24:24.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:24:24.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:24.153 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:24:24.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803811 2024-09-15T22:24:24.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803811 2024-09-15T22:24:24.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811' 2024-09-15T22:24:24.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:24.310 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:24:24.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574873 2024-09-15T22:24:24.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574873 2024-09-15T22:24:24.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574873' 2024-09-15T22:24:24.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:24.533 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:24:24.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378643 2024-09-15T22:24:24.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378643 2024-09-15T22:24:24.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574873 2-81604378643' 2024-09-15T22:24:24.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:24.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:24:24.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659139 2024-09-15T22:24:24.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659139 2024-09-15T22:24:24.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574873 2-81604378643 3-176093659139' 2024-09-15T22:24:24.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:24.834 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803811 2024-09-15T22:24:24.834 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:24.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:24:24.836 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803811 2024-09-15T22:24:24.836 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:24.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803811 2024-09-15T22:24:24.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-09-15T22:24:24.837 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803811 2024-09-15T22:24:24.837 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:24:25.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803811 -lt 25769803811 2024-09-15T22:24:25.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:25.274 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574873 2024-09-15T22:24:25.274 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:25.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:24:25.277 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574873 2024-09-15T22:24:25.277 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:25.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574873 2024-09-15T22:24:25.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574873' 2024-09-15T22:24:25.279 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 55834574873 2024-09-15T22:24:25.279 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:24:25.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574873 -lt 55834574873 2024-09-15T22:24:25.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:25.754 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378643 2024-09-15T22:24:25.754 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:25.756 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:24:25.757 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378643 2024-09-15T22:24:25.757 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:25.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378643 2024-09-15T22:24:25.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378643' 2024-09-15T22:24:25.759 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 81604378643 2024-09-15T22:24:25.760 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:24:26.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378643 -lt 81604378643 2024-09-15T22:24:26.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:26.235 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-176093659139 2024-09-15T22:24:26.235 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:26.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:24:26.238 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:26.238 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-176093659139 2024-09-15T22:24:26.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659139 2024-09-15T22:24:26.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 176093659139' 2024-09-15T22:24:26.239 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 176093659139 2024-09-15T22:24:26.239 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:24:26.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 176093659139 2024-09-15T22:24:26.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:24:27.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:24:27.713 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:24:28.148 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659139 -lt 176093659139 2024-09-15T22:24:28.149 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:24:28.149 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:24:28.149 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:24:28.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:24:28.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:24:28.702 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:24:28.702 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:24:28.702 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:24:28.702 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:24:28.702 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:24:28.703 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:24:29.143 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:24:29.143 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:24:29.144 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:24:29.144 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:24:29.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:24:29.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:24:29.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:24:29.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:24:29.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:24:30.407 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:24:30.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-0-10 2024-09-15T22:24:30.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:24:30.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:24:30.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-0-10 2024-09-15T22:24:30.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:24:30.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:24:30.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81867-0-10 td/test-erasure-eio/COPY 2024-09-15T22:24:30.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:24:30.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:24:30.622 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2024-09-15T22:24:30.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-81867-0-10 td/test-erasure-eio 1 10 set 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-0-10 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:24:30.625 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-0-10 2024-09-15T22:24:30.626 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:24:30.626 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-0-10 2024-09-15T22:24:30.627 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-0-10 2024-09-15T22:24:30.627 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:24:31.132 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:24:31.133 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:24:31.133 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:24:31.133 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:24:31.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:24:31.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:24:31.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-15T22:24:31.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:24:31.813 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:24:31.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-15T22:24:31.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-15T22:24:31.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-15T22:24:31.834 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records in 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records out 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:10 bytes copied, 0.000109561 s, 91.3 kB/s 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81867-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81867-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T22:24:31.835 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:24:31.836 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:24:31.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:24:31.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:24:31.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:24:32.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:24:32.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81867-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:32.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:24:32.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:24:32.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T22:24:32.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:24:32.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T22:24:32.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81867-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:34.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T22:24:34.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:24:34.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:24:34.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T22:24:34.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:24:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:24:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:24:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:24:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:24:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:24:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:24:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:24:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:24:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:24:34.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:24:34.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:24:34.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:24:34.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:24:34.848 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:24:34.848 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:24:34.848 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:24:34.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.1 2024-09-15T22:24:34.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:24:34.851 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T22:24:34.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T22:24:34.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:24:34.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:24:34.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:24:34.883 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:34.876+0000 7f43533197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:24:34.885 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:34.880+0000 7f43533197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:24:34.887 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:34.880+0000 7f43533197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:24:35.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T22:24:35.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:24:35.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:24:35.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:24:35.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:24:35.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:35.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:24:35.327 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:24:35.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:35.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:24:35.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:24:36.018 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:36.012+0000 7f43533197c0 -1 Falling back to public interface 2024-09-15T22:24:36.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:24:36.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:36.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:24:36.763 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:24:36.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:36.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:24:37.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:24:38.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:24:38.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:38.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:24:38.204 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:24:38.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:38.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:24:38.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:24:38.853 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:38.848+0000 7f43533197c0 -1 osd.1 47 log_to_monitors true 2024-09-15T22:24:39.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:24:39.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:39.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:24:39.630 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:24:39.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:39.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:24:40.246 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 up in weight 1 up_from 52 up_thru 52 down_at 48 last_clean_interval [13,47) [v2:127.0.0.1:6810/3478111472,v1:127.0.0.1:6811/3478111472] [v2:127.0.0.1:6812/3478111472,v1:127.0.0.1:6813/3478111472] exists,up b627b602-c3ed-4225-8c89-ee56604b494e 2024-09-15T22:24:40.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:24:40.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:24:40.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:24:40.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:24:40.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:24:40.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:24:40.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:24:40.247 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:24:40.248 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:24:40.248 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:24:40.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:24:40.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:24:40.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:24:40.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:24:40.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:24:40.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:24:40.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:24:40.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:24:40.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:24:40.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:24:40.792 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:24:40.792 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:24:40.792 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:24:40.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:24:40.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:40.793 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:24:41.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803816 2024-09-15T22:24:41.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803816 2024-09-15T22:24:41.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816' 2024-09-15T22:24:41.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:41.117 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:24:41.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=223338299395 2024-09-15T22:24:41.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 223338299395 2024-09-15T22:24:41.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816 1-223338299395' 2024-09-15T22:24:41.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:41.297 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:24:41.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378648 2024-09-15T22:24:41.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378648 2024-09-15T22:24:41.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816 1-223338299395 2-81604378648' 2024-09-15T22:24:41.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:41.449 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:24:41.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659144 2024-09-15T22:24:41.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659144 2024-09-15T22:24:41.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816 1-223338299395 2-81604378648 3-176093659144' 2024-09-15T22:24:41.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:41.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803816 2024-09-15T22:24:41.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:41.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:24:41.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803816 2024-09-15T22:24:41.602 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:41.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803816 2024-09-15T22:24:41.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-09-15T22:24:41.603 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803816 2024-09-15T22:24:41.603 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:24:42.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803814 -lt 25769803816 2024-09-15T22:24:42.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:24:43.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:24:43.018 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:24:43.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803817 -lt 25769803816 2024-09-15T22:24:43.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:43.440 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-223338299395 2024-09-15T22:24:43.440 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:43.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:24:43.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-223338299395 2024-09-15T22:24:43.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:43.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=223338299395 2024-09-15T22:24:43.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 223338299395' 2024-09-15T22:24:43.443 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 223338299395 2024-09-15T22:24:43.444 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:24:43.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 223338299395 -lt 223338299395 2024-09-15T22:24:43.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:43.938 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:43.938 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378648 2024-09-15T22:24:43.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:24:43.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378648 2024-09-15T22:24:43.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:43.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378648 2024-09-15T22:24:43.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378648' 2024-09-15T22:24:43.941 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 81604378648 2024-09-15T22:24:43.942 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:24:44.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378649 -lt 81604378648 2024-09-15T22:24:44.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:44.439 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-176093659144 2024-09-15T22:24:44.439 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:44.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:24:44.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:44.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-176093659144 2024-09-15T22:24:44.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659144 2024-09-15T22:24:44.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 176093659144' 2024-09-15T22:24:44.443 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 176093659144 2024-09-15T22:24:44.443 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:24:44.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659145 -lt 176093659144 2024-09-15T22:24:44.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:24:44.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:24:44.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:24:45.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:24:45.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:24:45.475 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:24:45.475 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:24:45.475 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:24:45.476 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:24:45.476 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:24:45.476 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:24:45.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:24:45.915 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:24:45.915 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:24:45.916 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:24:46.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:24:46.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:24:46.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:24:46.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:24:46.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:24:47.107 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:24:47.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-15T22:24:47.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-0-10 fail 2024-09-15T22:24:47.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:24:47.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:24:47.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-0-10 2024-09-15T22:24:47.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T22:24:47.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T22:24:47.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81867-0-10 td/test-erasure-eio/COPY 2024-09-15T22:24:47.283 INFO:tasks.workunit.client.0.smithi104.stderr:error getting pool-jerasure/obj-size-81867-0-10: (5) Input/output error 2024-09-15T22:24:47.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T22:24:47.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T22:24:47.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:330: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2024-09-15T22:24:47.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T22:24:47.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-15T22:24:47.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=0 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-81867-0-0 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81867-0-0 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81867-0-0 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:24:47.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:24:47.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:24:47.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:24:47.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:24:47.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:24:47.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81867-0-0 td/test-erasure-eio/ORIGINAL 2024-09-15T22:24:47.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-81867-0-0 td/test-erasure-eio 0 0 set 2024-09-15T22:24:47.506 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-0-0 2024-09-15T22:24:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:24:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:24:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:24:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2024-09-15T22:24:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:24:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-15T22:24:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:24:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-15T22:24:47.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:24:47.508 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-0-0 2024-09-15T22:24:47.508 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:24:47.508 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-0-0 2024-09-15T22:24:47.508 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-0-0 2024-09-15T22:24:47.509 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:24:47.972 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:24:47.972 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:24:47.972 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:24:47.972 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:24:47.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:24:47.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:24:47.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2024-09-15T22:24:47.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:24:48.645 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:24:48.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-15T22:24:48.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-15T22:24:48.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-15T22:24:48.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81867-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:48.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:24:48.665 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:24:48.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T22:24:48.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:24:48.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81867-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:48.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:24:48.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:24:48.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T22:24:48.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:24:48.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-15T22:24:48.667 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:24:48.667 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:24:48.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:24:48.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:24:48.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:24:49.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:24:49.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81867-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:49.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:24:49.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:24:49.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T22:24:49.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:24:49.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-15T22:24:49.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81867-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:24:52.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-15T22:24:52.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:24:52.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:24:52.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T22:24:52.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:24:52.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:24:52.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:24:52.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:24:52.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:24:52.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:24:52.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:24:52.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:24:52.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:24:52.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:24:52.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:24:52.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:24:52.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:24:52.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:24:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:24:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:24:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:24:52.056 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:24:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:24:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:24:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:24:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:24:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:24:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:24:52.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:24:52.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:24:52.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:24:52.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:24:52.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:24:52.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T22:24:52.057 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.3 2024-09-15T22:24:52.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:24:52.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-15T22:24:52.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T22:24:52.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:24:52.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:24:52.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:24:52.091 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:52.084+0000 7f24ca2357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:24:52.094 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:52.088+0000 7f24ca2357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:24:52.096 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:52.092+0000 7f24ca2357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:24:52.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T22:24:52.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:24:52.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:24:52.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:24:52.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:24:52.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:52.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:24:52.501 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:24:52.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:24:52.501 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:52.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:24:53.122 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:53.116+0000 7f24ca2357c0 -1 Falling back to public interface 2024-09-15T22:24:53.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:24:53.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:53.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:24:53.976 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:24:53.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:53.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:24:54.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:24:55.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:24:55.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:55.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:24:55.412 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:24:55.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:55.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:24:55.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:24:55.847 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:24:55.844+0000 7f24ca2357c0 -1 osd.3 58 log_to_monitors true 2024-09-15T22:24:56.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:24:56.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:24:56.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:24:56.841 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:24:56.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:24:56.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:24:57.394 INFO:tasks.workunit.client.0.smithi104.stderr:osd.3 up in weight 1 up_from 62 up_thru 62 down_at 59 last_clean_interval [41,58) [v2:127.0.0.1:6826/1404564646,v1:127.0.0.1:6827/1404564646] [v2:127.0.0.1:6828/1404564646,v1:127.0.0.1:6829/1404564646] exists,up 83950c7b-64b0-45d4-afcc-9d0c8c314315 2024-09-15T22:24:57.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:24:57.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:24:57.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:24:57.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:24:57.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:24:57.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:24:57.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:24:57.395 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:24:57.396 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:24:57.396 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:24:57.396 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:24:57.396 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:24:57.397 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:24:57.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:24:57.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:24:57.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:24:57.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:24:57.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:24:57.503 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:24:58.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:24:58.033 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:24:58.033 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:24:58.033 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:24:58.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:24:58.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:58.033 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:24:58.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803822 2024-09-15T22:24:58.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803822 2024-09-15T22:24:58.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803822' 2024-09-15T22:24:58.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:58.299 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:24:58.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=223338299400 2024-09-15T22:24:58.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 223338299400 2024-09-15T22:24:58.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803822 1-223338299400' 2024-09-15T22:24:58.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:58.459 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:24:58.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378654 2024-09-15T22:24:58.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378654 2024-09-15T22:24:58.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803822 1-223338299400 2-81604378654' 2024-09-15T22:24:58.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:24:58.614 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:24:58.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=266287972355 2024-09-15T22:24:58.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 266287972355 2024-09-15T22:24:58.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803822 1-223338299400 2-81604378654 3-266287972355' 2024-09-15T22:24:58.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:58.758 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803822 2024-09-15T22:24:58.758 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:58.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:24:58.760 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803822 2024-09-15T22:24:58.760 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:58.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803822 2024-09-15T22:24:58.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803822' 2024-09-15T22:24:58.761 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803822 2024-09-15T22:24:58.762 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:24:59.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803822 -lt 25769803822 2024-09-15T22:24:59.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:59.176 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-223338299400 2024-09-15T22:24:59.176 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:59.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:24:59.178 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-223338299400 2024-09-15T22:24:59.178 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:59.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=223338299400 2024-09-15T22:24:59.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 223338299400' 2024-09-15T22:24:59.180 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 223338299400 2024-09-15T22:24:59.180 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:24:59.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 223338299400 -lt 223338299400 2024-09-15T22:24:59.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:24:59.595 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378654 2024-09-15T22:24:59.595 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:24:59.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:24:59.597 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378654 2024-09-15T22:24:59.597 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:24:59.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378654 2024-09-15T22:24:59.599 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378654' 2024-09-15T22:24:59.599 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 81604378654 2024-09-15T22:24:59.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:25:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378654 -lt 81604378654 2024-09-15T22:25:00.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:00.030 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-266287972355 2024-09-15T22:25:00.030 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:00.031 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:25:00.031 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-266287972355 2024-09-15T22:25:00.031 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:00.032 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=266287972355 2024-09-15T22:25:00.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 266287972355' 2024-09-15T22:25:00.033 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 266287972355 2024-09-15T22:25:00.033 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:25:00.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 266287972355 2024-09-15T22:25:00.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:25:01.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:25:01.516 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:25:01.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 266287972355 -lt 266287972355 2024-09-15T22:25:01.948 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:25:01.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:25:01.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:25:02.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:25:02.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:25:02.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:25:02.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:25:02.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:25:02.518 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:25:02.519 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:25:02.519 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:25:02.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:25:02.931 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:25:02.932 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:25:02.932 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:25:03.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:25:03.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:25:03.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:25:03.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:25:03.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:25:04.125 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:25:04.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-0-0 2024-09-15T22:25:04.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:25:04.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:25:04.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-0-0 2024-09-15T22:25:04.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:25:04.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:25:04.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81867-0-0 td/test-erasure-eio/COPY 2024-09-15T22:25:04.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:25:04.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:25:04.301 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2024-09-15T22:25:04.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2024-09-15T22:25:04.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-81867-0-0 td/test-erasure-eio 1 0 set 2024-09-15T22:25:04.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-0-0 2024-09-15T22:25:04.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:25:04.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:25:04.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:25:04.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-15T22:25:04.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:25:04.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-15T22:25:04.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:25:04.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-15T22:25:04.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:25:04.305 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-0-0 2024-09-15T22:25:04.305 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:25:04.305 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-0-0 2024-09-15T22:25:04.305 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-0-0 2024-09-15T22:25:04.305 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:25:04.742 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:25:04.743 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:25:04.743 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:25:04.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:25:04.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:25:04.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:25:04.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-15T22:25:04.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:25:05.573 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:25:05.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-15T22:25:05.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-15T22:25:05.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-15T22:25:05.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81867-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:05.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:25:05.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:25:05.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T22:25:05.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:25:05.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81867-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:05.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:25:05.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:25:05.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T22:25:05.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:25:05.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T22:25:05.597 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:25:05.597 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:25:05.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:25:05.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:25:05.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:25:06.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:25:06.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81867-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:06.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:25:06.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:25:06.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T22:25:06.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:25:06.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T22:25:06.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81867-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:08.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T22:25:08.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:25:08.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:25:08.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T22:25:08.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:25:08.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:25:08.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:25:08.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:25:08.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:25:08.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:25:08.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:25:08.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:25:08.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:25:08.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:25:08.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:25:08.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:25:08.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:25:08.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:25:08.882 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:25:08.883 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:25:08.883 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:25:08.883 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:25:08.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:25:08.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:25:08.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:25:08.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:25:08.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:25:08.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:25:08.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:25:08.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:25:08.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:25:08.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:25:08.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:25:08.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T22:25:08.886 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.1 2024-09-15T22:25:08.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:25:08.886 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T22:25:08.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T22:25:08.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:25:08.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:25:08.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:25:08.917 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:08.916+0000 7f3ab49e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:08.920 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:08.920+0000 7f3ab49e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:08.921 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:08.920+0000 7f3ab49e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:09.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T22:25:09.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:25:09.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:25:09.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:25:09.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:25:09.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:09.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:25:09.322 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:25:09.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:09.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:25:09.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:10.174 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:10.172+0000 7f3ab49e67c0 -1 Falling back to public interface 2024-09-15T22:25:10.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:10.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:10.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:25:10.803 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:25:10.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:10.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:25:11.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:12.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:12.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:12.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:25:12.269 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:25:12.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:12.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:25:12.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:13.074 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:13.071+0000 7f3ab49e67c0 -1 osd.1 67 log_to_monitors true 2024-09-15T22:25:13.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:13.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:13.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:25:13.696 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:25:13.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:13.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:25:14.253 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [52,67) [v2:127.0.0.1:6810/4240693909,v1:127.0.0.1:6811/4240693909] [v2:127.0.0.1:6812/4240693909,v1:127.0.0.1:6813/4240693909] exists,up b627b602-c3ed-4225-8c89-ee56604b494e 2024-09-15T22:25:14.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:25:14.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:25:14.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:25:14.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:25:14.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:25:14.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:25:14.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:25:14.254 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:25:14.255 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:25:14.255 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:25:14.255 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:25:14.255 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:25:14.255 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:25:14.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:25:14.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:25:14.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:25:14.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:25:14.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:25:14.389 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:25:15.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:25:15.071 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:25:15.071 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:25:15.071 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:25:15.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:25:15.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:15.071 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:25:15.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803827 2024-09-15T22:25:15.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803827 2024-09-15T22:25:15.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827' 2024-09-15T22:25:15.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:15.313 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:25:15.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645315 2024-09-15T22:25:15.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645315 2024-09-15T22:25:15.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-309237645315' 2024-09-15T22:25:15.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:15.474 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:25:15.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378659 2024-09-15T22:25:15.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378659 2024-09-15T22:25:15.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-309237645315 2-81604378659' 2024-09-15T22:25:15.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:15.624 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:25:15.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=266287972360 2024-09-15T22:25:15.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 266287972360 2024-09-15T22:25:15.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-309237645315 2-81604378659 3-266287972360' 2024-09-15T22:25:15.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:15.765 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803827 2024-09-15T22:25:15.765 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:15.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:25:15.767 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803827 2024-09-15T22:25:15.767 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:15.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803827 2024-09-15T22:25:15.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-09-15T22:25:15.768 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803827 2024-09-15T22:25:15.769 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:25:16.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803825 -lt 25769803827 2024-09-15T22:25:16.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:25:17.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:25:17.194 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:25:17.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803828 -lt 25769803827 2024-09-15T22:25:17.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:17.614 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:17.614 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-309237645315 2024-09-15T22:25:17.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:25:17.616 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-309237645315 2024-09-15T22:25:17.616 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:17.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645315 2024-09-15T22:25:17.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 309237645315' 2024-09-15T22:25:17.617 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 309237645315 2024-09-15T22:25:17.617 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:25:18.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645315 -lt 309237645315 2024-09-15T22:25:18.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:18.044 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378659 2024-09-15T22:25:18.044 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:18.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:25:18.046 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378659 2024-09-15T22:25:18.046 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:18.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378659 2024-09-15T22:25:18.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378659' 2024-09-15T22:25:18.047 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 81604378659 2024-09-15T22:25:18.047 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:25:18.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378659 -lt 81604378659 2024-09-15T22:25:18.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:18.461 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-266287972360 2024-09-15T22:25:18.462 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:18.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:25:18.465 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-266287972360 2024-09-15T22:25:18.465 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:18.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=266287972360 2024-09-15T22:25:18.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 266287972360' 2024-09-15T22:25:18.466 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 266287972360 2024-09-15T22:25:18.466 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:25:18.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 266287972361 -lt 266287972360 2024-09-15T22:25:18.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:25:18.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:25:18.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:25:19.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:25:19.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:25:19.519 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:25:19.519 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:25:19.519 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:25:19.519 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:25:19.519 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:25:19.520 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:25:19.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:25:19.920 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:25:19.920 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:25:19.920 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:25:20.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:25:20.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:25:20.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:25:20.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:25:20.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:25:21.178 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:25:21.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-15T22:25:21.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-0-0 fail 2024-09-15T22:25:21.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:25:21.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:25:21.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-0-0 2024-09-15T22:25:21.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T22:25:21.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T22:25:21.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81867-0-0 td/test-erasure-eio/COPY 2024-09-15T22:25:21.348 INFO:tasks.workunit.client.0.smithi104.stderr:error getting pool-jerasure/obj-size-81867-0-0: (5) Input/output error 2024-09-15T22:25:21.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T22:25:21.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T22:25:21.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:331: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2024-09-15T22:25:21.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T22:25:21.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-15T22:25:21.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2024-09-15T22:25:21.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=256 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=add 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-81867-0-256 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81867-0-256 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81867-0-256 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:25:21.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:25:21.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:25:21.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:25:21.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:25:21.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:25:21.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81867-0-256 td/test-erasure-eio/ORIGINAL 2024-09-15T22:25:21.568 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-81867-0-256 td/test-erasure-eio 0 256 add 2024-09-15T22:25:21.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-0-256 2024-09-15T22:25:21.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:25:21.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:25:21.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:25:21.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2024-09-15T22:25:21.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:25:21.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-15T22:25:21.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:25:21.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-15T22:25:21.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:25:21.570 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-0-256 2024-09-15T22:25:21.570 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:25:21.570 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-0-256 2024-09-15T22:25:21.570 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:25:21.570 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-0-256 2024-09-15T22:25:22.028 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:25:22.028 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:25:22.028 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:25:22.028 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:25:22.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:25:22.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:25:22.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2024-09-15T22:25:22.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:25:22.683 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:25:22.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-15T22:25:22.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81867-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:22.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:25:22.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:25:22.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T22:25:22.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:25:22.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81867-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:22.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:25:22.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:25:22.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T22:25:22.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:25:22.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-15T22:25:22.705 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:25:22.705 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:25:22.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:25:22.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:25:22.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:25:24.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:25:24.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81867-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:24.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:25:24.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:25:24.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T22:25:24.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:25:24.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-15T22:25:24.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81867-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:24.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-15T22:25:24.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:25:24.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:25:24.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T22:25:24.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:25:24.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:25:24.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:25:24.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:25:24.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:25:24.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:25:24.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:25:24.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:25:24.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:25:24.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:25:24.707 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:25:24.709 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:25:24.709 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:25:24.709 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:25:24.709 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:25:24.709 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:25:24.709 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:25:24.709 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:25:24.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:25:24.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:25:24.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:25:24.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:25:24.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:25:24.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:25:24.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:25:24.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:25:24.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:25:24.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:25:24.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:25:24.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T22:25:24.710 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.3 2024-09-15T22:25:24.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:25:24.711 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-15T22:25:24.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T22:25:24.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:25:24.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:25:24.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:25:24.752 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:24.751+0000 7fefb127e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:24.783 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:24.779+0000 7fefb127e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:24.784 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:24.783+0000 7fefb127e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:25.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T22:25:25.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:25:25.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:25:25.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:25:25.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:25:25.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:25.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:25:25.168 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:25:25.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:25:25.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:25.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:25.826 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:25.823+0000 7fefb127e7c0 -1 Falling back to public interface 2024-09-15T22:25:26.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:26.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:26.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:25:26.597 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:25:26.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:26.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:25:27.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:28.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:28.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:28.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:25:28.070 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:25:28.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:28.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:25:28.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:29.100 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:29.099+0000 7fefb127e7c0 -1 osd.3 77 log_to_monitors true 2024-09-15T22:25:29.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:29.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:29.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:25:29.511 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:25:29.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:29.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:25:30.176 INFO:tasks.workunit.client.0.smithi104.stderr:osd.3 up in weight 1 up_from 83 up_thru 83 down_at 79 last_clean_interval [62,77) [v2:127.0.0.1:6826/4097542770,v1:127.0.0.1:6827/4097542770] [v2:127.0.0.1:6828/4097542770,v1:127.0.0.1:6829/4097542770] exists,up 83950c7b-64b0-45d4-afcc-9d0c8c314315 2024-09-15T22:25:30.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:25:30.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:25:30.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:25:30.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:25:30.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:25:30.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:25:30.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:25:30.177 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:25:30.177 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:25:30.177 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:25:30.178 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:25:30.178 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:25:30.178 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:25:30.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:25:30.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:25:30.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:25:30.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:25:30.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:25:30.282 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:25:30.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:25:30.758 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:25:30.758 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:25:30.759 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:25:30.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:25:30.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:30.759 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:25:30.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803832 2024-09-15T22:25:30.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803832 2024-09-15T22:25:30.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832' 2024-09-15T22:25:30.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:30.912 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:25:31.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645320 2024-09-15T22:25:31.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645320 2024-09-15T22:25:31.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-309237645320' 2024-09-15T22:25:31.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:31.085 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:25:31.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378664 2024-09-15T22:25:31.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378664 2024-09-15T22:25:31.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-309237645320 2-81604378664' 2024-09-15T22:25:31.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:31.304 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:25:31.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285571 2024-09-15T22:25:31.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285571 2024-09-15T22:25:31.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-309237645320 2-81604378664 3-356482285571' 2024-09-15T22:25:31.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:31.450 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:31.450 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803832 2024-09-15T22:25:31.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:25:31.452 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803832 2024-09-15T22:25:31.452 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:31.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803832 2024-09-15T22:25:31.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803832' 2024-09-15T22:25:31.453 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803832 2024-09-15T22:25:31.453 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:25:31.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803830 -lt 25769803832 2024-09-15T22:25:31.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:25:32.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:25:32.914 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:25:33.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803833 -lt 25769803832 2024-09-15T22:25:33.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:33.344 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-309237645320 2024-09-15T22:25:33.344 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:33.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:25:33.346 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-309237645320 2024-09-15T22:25:33.347 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:33.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645320 2024-09-15T22:25:33.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 309237645320' 2024-09-15T22:25:33.348 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 309237645320 2024-09-15T22:25:33.348 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:25:33.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645320 -lt 309237645320 2024-09-15T22:25:33.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:33.816 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378664 2024-09-15T22:25:33.817 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:33.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:25:33.819 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378664 2024-09-15T22:25:33.819 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:33.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378664 2024-09-15T22:25:33.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378664' 2024-09-15T22:25:33.821 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 81604378664 2024-09-15T22:25:33.821 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:25:34.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378665 -lt 81604378664 2024-09-15T22:25:34.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:34.277 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-356482285571 2024-09-15T22:25:34.277 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:34.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:25:34.280 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-356482285571 2024-09-15T22:25:34.280 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:34.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285571 2024-09-15T22:25:34.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 356482285571' 2024-09-15T22:25:34.281 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 356482285571 2024-09-15T22:25:34.281 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:25:34.751 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285571 -lt 356482285571 2024-09-15T22:25:34.752 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:25:34.752 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:25:34.752 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:25:35.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:25:35.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:25:35.298 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:25:35.298 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:25:35.298 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:25:35.298 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:25:35.298 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:25:35.298 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:25:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:25:35.704 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:25:35.704 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:25:35.704 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:25:36.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:25:36.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:25:36.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:25:36.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-15T22:25:36.291 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records in 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records out 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:256 bytes copied, 4.4101e-05 s, 5.8 MB/s 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81867-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81867-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T22:25:36.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:25:36.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-15T22:25:36.293 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:25:36.293 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:25:36.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:25:36.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:25:36.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:25:36.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:25:36.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81867-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:36.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:25:36.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:25:36.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T22:25:36.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:25:36.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-15T22:25:36.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81867-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:38.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:25:38.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:25:38.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:25:38.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:25:38.913 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:25:38.913 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:25:38.913 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:25:38.914 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:25:38.914 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:25:38.914 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:25:38.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:25:38.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T22:25:38.916 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.3 2024-09-15T22:25:38.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:25:38.916 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-15T22:25:38.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T22:25:38.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:25:38.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:25:38.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:25:38.949 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:38.947+0000 7f77015e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:38.952 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:38.951+0000 7f77015e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:38.953 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:38.951+0000 7f77015e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:39.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T22:25:39.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:25:39.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:25:39.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:25:39.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:25:39.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:39.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:25:39.428 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:25:39.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:39.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:25:39.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:40.013 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:40.011+0000 7f77015e67c0 -1 Falling back to public interface 2024-09-15T22:25:40.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:40.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:40.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:25:40.920 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:25:40.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:40.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:25:41.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:42.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:42.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:42.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:25:42.349 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:25:42.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:42.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:25:42.597 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:42.595+0000 7f77015e67c0 -1 osd.3 84 log_to_monitors true 2024-09-15T22:25:42.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:43.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:43.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:43.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:25:43.825 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:25:43.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:43.826 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:25:44.248 INFO:tasks.workunit.client.0.smithi104.stderr:osd.3 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [83,84) [v2:127.0.0.1:6826/3372604683,v1:127.0.0.1:6827/3372604683] [v2:127.0.0.1:6828/3372604683,v1:127.0.0.1:6829/3372604683] exists,up 83950c7b-64b0-45d4-afcc-9d0c8c314315 2024-09-15T22:25:44.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:25:44.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:25:44.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:25:44.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:25:44.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:25:44.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:25:44.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:25:44.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:25:44.249 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:25:44.249 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:25:44.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:25:44.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:25:44.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:25:44.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:25:44.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:25:44.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:25:44.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:25:44.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:25:44.358 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:25:44.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:25:44.785 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:25:44.785 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:25:44.785 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:25:44.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:25:44.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:44.785 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:25:44.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803837 2024-09-15T22:25:44.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803837 2024-09-15T22:25:44.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803837' 2024-09-15T22:25:44.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:44.924 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:25:45.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645325 2024-09-15T22:25:45.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645325 2024-09-15T22:25:45.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803837 1-309237645325' 2024-09-15T22:25:45.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:45.068 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:25:45.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378669 2024-09-15T22:25:45.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378669 2024-09-15T22:25:45.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803837 1-309237645325 2-81604378669' 2024-09-15T22:25:45.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:25:45.211 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:25:45.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122051 2024-09-15T22:25:45.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122051 2024-09-15T22:25:45.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803837 1-309237645325 2-81604378669 3-377957122051' 2024-09-15T22:25:45.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:45.354 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803837 2024-09-15T22:25:45.354 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:45.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:25:45.355 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803837 2024-09-15T22:25:45.355 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:45.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803837 2024-09-15T22:25:45.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803837' 2024-09-15T22:25:45.356 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803837 2024-09-15T22:25:45.357 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:25:45.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803835 -lt 25769803837 2024-09-15T22:25:45.754 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:25:46.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:25:46.755 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:25:47.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803838 -lt 25769803837 2024-09-15T22:25:47.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:47.346 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-309237645325 2024-09-15T22:25:47.346 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:47.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:25:47.349 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-309237645325 2024-09-15T22:25:47.349 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:47.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645325 2024-09-15T22:25:47.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 309237645325' 2024-09-15T22:25:47.350 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 309237645325 2024-09-15T22:25:47.351 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:25:47.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645325 -lt 309237645325 2024-09-15T22:25:47.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:47.834 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378669 2024-09-15T22:25:47.834 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:47.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:25:47.836 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378669 2024-09-15T22:25:47.837 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:47.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378669 2024-09-15T22:25:47.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378669' 2024-09-15T22:25:47.838 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 81604378669 2024-09-15T22:25:47.838 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:25:48.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378669 -lt 81604378669 2024-09-15T22:25:48.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:25:48.334 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-377957122051 2024-09-15T22:25:48.334 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:25:48.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:25:48.337 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-377957122051 2024-09-15T22:25:48.337 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:25:48.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122051 2024-09-15T22:25:48.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 377957122051' 2024-09-15T22:25:48.338 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 377957122051 2024-09-15T22:25:48.339 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:25:48.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122051 -lt 377957122051 2024-09-15T22:25:48.803 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:25:48.804 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:25:48.804 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:25:49.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:25:49.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:25:49.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:25:49.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:25:49.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:25:49.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:25:49.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:25:49.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:25:49.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:25:49.837 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:25:49.838 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:25:49.838 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:25:50.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:25:50.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:25:50.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:25:50.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:25:50.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:25:50.986 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:25:51.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-0-256 2024-09-15T22:25:51.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:25:51.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:25:51.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-0-256 2024-09-15T22:25:51.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:25:51.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:25:51.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81867-0-256 td/test-erasure-eio/COPY 2024-09-15T22:25:51.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:25:51.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:25:51.205 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-81867-0-256 td/test-erasure-eio 1 256 add 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-0-256 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-15T22:25:51.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:25:51.208 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-0-256 2024-09-15T22:25:51.208 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:25:51.208 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-0-256 2024-09-15T22:25:51.209 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-0-256 2024-09-15T22:25:51.209 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:25:51.632 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:25:51.633 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:25:51.633 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:25:51.633 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:25:51.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:25:51.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:25:51.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-15T22:25:51.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:25:52.308 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:25:52.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-15T22:25:52.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81867-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:52.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:25:52.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:25:52.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T22:25:52.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:25:52.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81867-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:52.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:25:52.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:25:52.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T22:25:52.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:25:52.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T22:25:52.330 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:25:52.330 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:25:52.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:25:52.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:25:52.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:25:52.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:25:52.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81867-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:52.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:25:52.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:25:52.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T22:25:52.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:25:52.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T22:25:52.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81867-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:25:53.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T22:25:53.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:25:53.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:25:53.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T22:25:53.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:25:53.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:25:53.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:25:53.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:25:53.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:25:53.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:25:53.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:25:53.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:25:53.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:25:53.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:25:53.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:25:53.459 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:25:53.459 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:25:53.459 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:25:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:25:53.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:25:53.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:25:53.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:25:53.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T22:25:53.462 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.1 2024-09-15T22:25:53.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:25:53.463 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T22:25:53.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T22:25:53.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:25:53.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:25:53.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:25:53.496 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:53.495+0000 7fbdbdd0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:53.624 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:53.623+0000 7fbdbdd0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:53.625 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:53.623+0000 7fbdbdd0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:25:54.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T22:25:54.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:25:54.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:25:54.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:25:54.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:25:54.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:54.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:25:54.060 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:25:54.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:54.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:25:54.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:54.915 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:54.911+0000 7fbdbdd0b7c0 -1 Falling back to public interface 2024-09-15T22:25:55.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:55.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:55.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:25:55.526 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:25:55.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:55.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:25:56.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:57.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:57.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:57.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:25:57.011 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:25:57.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:57.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:25:57.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:25:58.068 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:25:58.067+0000 7fbdbdd0b7c0 -1 osd.1 91 log_to_monitors true 2024-09-15T22:25:58.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:25:58.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:25:58.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:25:58.447 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:25:58.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:25:58.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:25:59.084 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 up in weight 1 up_from 98 up_thru 85 down_at 94 last_clean_interval [72,91) [v2:127.0.0.1:6810/3992234658,v1:127.0.0.1:6811/3992234658] [v2:127.0.0.1:6812/3992234658,v1:127.0.0.1:6813/3992234658] exists,up b627b602-c3ed-4225-8c89-ee56604b494e 2024-09-15T22:25:59.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:25:59.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:25:59.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:25:59.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:25:59.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:25:59.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:25:59.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:25:59.085 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:25:59.086 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:25:59.086 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:25:59.086 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:25:59.086 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:25:59.087 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:25:59.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:25:59.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:25:59.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:25:59.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:25:59.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:25:59.205 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:26:00.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:26:00.007 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:26:00.007 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:26:00.007 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:26:00.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:26:00.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:26:00.008 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:26:00.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803842 2024-09-15T22:26:00.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803842 2024-09-15T22:26:00.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842' 2024-09-15T22:26:00.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:26:00.214 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:26:00.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795011 2024-09-15T22:26:00.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795011 2024-09-15T22:26:00.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-420906795011' 2024-09-15T22:26:00.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:26:00.418 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:26:00.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378674 2024-09-15T22:26:00.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378674 2024-09-15T22:26:00.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-420906795011 2-81604378674' 2024-09-15T22:26:00.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:26:00.576 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:26:00.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122056 2024-09-15T22:26:00.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122056 2024-09-15T22:26:00.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-420906795011 2-81604378674 3-377957122056' 2024-09-15T22:26:00.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:26:00.728 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803842 2024-09-15T22:26:00.728 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:26:00.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:26:00.730 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803842 2024-09-15T22:26:00.730 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:26:00.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803842 2024-09-15T22:26:00.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803842' 2024-09-15T22:26:00.731 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803842 2024-09-15T22:26:00.732 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:26:01.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803842 -lt 25769803842 2024-09-15T22:26:01.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:26:01.243 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-420906795011 2024-09-15T22:26:01.243 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:26:01.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:26:01.245 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-420906795011 2024-09-15T22:26:01.245 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:26:01.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795011 2024-09-15T22:26:01.247 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 420906795011' 2024-09-15T22:26:01.247 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 420906795011 2024-09-15T22:26:01.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:26:01.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795011 -lt 420906795011 2024-09-15T22:26:01.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:26:01.729 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:26:01.729 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378674 2024-09-15T22:26:01.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:26:01.731 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378674 2024-09-15T22:26:01.731 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:26:01.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378674 2024-09-15T22:26:01.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378674' 2024-09-15T22:26:01.733 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 81604378674 2024-09-15T22:26:01.733 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:26:02.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378674 -lt 81604378674 2024-09-15T22:26:02.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:26:02.163 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-377957122056 2024-09-15T22:26:02.163 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:26:02.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:26:02.165 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-377957122056 2024-09-15T22:26:02.165 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:26:02.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122056 2024-09-15T22:26:02.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 377957122056' 2024-09-15T22:26:02.167 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 377957122056 2024-09-15T22:26:02.167 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:26:02.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122055 -lt 377957122056 2024-09-15T22:26:02.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:26:03.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:26:03.631 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:26:04.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122056 -lt 377957122056 2024-09-15T22:26:04.089 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:26:04.090 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:26:04.090 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:26:04.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:26:04.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:26:04.660 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:26:04.660 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:26:04.661 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:26:04.661 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:26:04.664 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:26:04.664 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:26:05.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:26:05.107 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:26:05.108 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:26:05.108 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:26:05.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:26:05.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:26:05.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:26:05.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-15T22:26:05.642 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records in 2024-09-15T22:26:05.642 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records out 2024-09-15T22:26:05.642 INFO:tasks.workunit.client.0.smithi104.stderr:256 bytes copied, 4.0286e-05 s, 6.4 MB/s 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81867-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81867-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T22:26:05.643 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:26:05.644 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:26:05.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:26:05.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:26:05.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:26:05.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:26:05.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81867-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:26:05.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:26:05.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:26:05.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T22:26:05.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:26:05.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T22:26:05.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81867-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:26:08.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T22:26:08.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:26:08.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:26:08.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T22:26:08.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:26:08.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:26:08.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:26:08.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:26:08.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:26:08.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:26:08.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:26:08.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:26:08.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:26:08.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:26:08.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:26:08.807 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:26:08.807 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:26:08.807 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:26:08.809 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:26:08.809 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:26:08.809 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:26:08.809 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:26:08.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:26:08.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:26:08.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:26:08.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:26:08.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:26:08.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:26:08.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:26:08.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:26:08.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:26:08.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:26:08.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:26:08.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T22:26:08.811 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.1 2024-09-15T22:26:08.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:26:08.812 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T22:26:08.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T22:26:08.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:26:08.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:26:08.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:26:08.844 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:08.843+0000 7f7c33b717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:26:08.871 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:08.867+0000 7f7c33b717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:26:08.872 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:08.871+0000 7f7c33b717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:26:09.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T22:26:09.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:26:09.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:26:09.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:26:09.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:26:09.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:09.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:26:09.308 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:26:09.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:26:09.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:09.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:10.142 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:10.139+0000 7f7c33b717c0 -1 Falling back to public interface 2024-09-15T22:26:10.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:10.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:10.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:26:10.780 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:26:10.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:10.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:26:11.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:12.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:12.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:12.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:26:12.263 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:26:12.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:12.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:26:12.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:12.891 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:12.891+0000 7f7c33b717c0 -1 osd.1 100 log_to_monitors true 2024-09-15T22:26:13.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:13.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:13.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:26:13.686 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:26:13.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:13.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:26:14.325 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 up in weight 1 up_from 105 up_thru 105 down_at 101 last_clean_interval [98,100) [v2:127.0.0.1:6810/2286459535,v1:127.0.0.1:6811/2286459535] [v2:127.0.0.1:6812/2286459535,v1:127.0.0.1:6813/2286459535] exists,up b627b602-c3ed-4225-8c89-ee56604b494e 2024-09-15T22:26:14.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:26:14.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:26:14.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:26:14.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:26:14.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:26:14.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:26:14.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:26:14.326 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:26:14.328 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:26:14.328 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:26:14.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:26:14.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:26:14.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:26:14.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:26:14.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:26:14.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:26:14.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:26:14.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:26:14.448 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:26:15.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:26:15.094 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:26:15.094 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:26:15.094 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:26:15.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:26:15.094 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:26:15.095 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:26:15.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803847 2024-09-15T22:26:15.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803847 2024-09-15T22:26:15.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847' 2024-09-15T22:26:15.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:26:15.252 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:26:15.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566083 2024-09-15T22:26:15.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566083 2024-09-15T22:26:15.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847 1-450971566083' 2024-09-15T22:26:15.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:26:15.403 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:26:15.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378679 2024-09-15T22:26:15.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378679 2024-09-15T22:26:15.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847 1-450971566083 2-81604378679' 2024-09-15T22:26:15.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:26:15.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:26:15.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122061 2024-09-15T22:26:15.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122061 2024-09-15T22:26:15.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847 1-450971566083 2-81604378679 3-377957122061' 2024-09-15T22:26:15.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:26:15.703 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803847 2024-09-15T22:26:15.703 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:26:15.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:26:15.704 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803847 2024-09-15T22:26:15.704 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:26:15.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803847 2024-09-15T22:26:15.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803847' 2024-09-15T22:26:15.705 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803847 2024-09-15T22:26:15.706 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:26:16.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803845 -lt 25769803847 2024-09-15T22:26:16.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:26:17.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:26:17.116 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:26:17.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803848 -lt 25769803847 2024-09-15T22:26:17.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:26:17.590 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-450971566083 2024-09-15T22:26:17.591 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:26:17.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:26:17.593 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-450971566083 2024-09-15T22:26:17.593 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:26:17.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566083 2024-09-15T22:26:17.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 450971566083' 2024-09-15T22:26:17.595 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 450971566083 2024-09-15T22:26:17.595 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:26:18.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566083 -lt 450971566083 2024-09-15T22:26:18.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:26:18.088 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:26:18.088 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378679 2024-09-15T22:26:18.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:26:18.090 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378679 2024-09-15T22:26:18.090 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:26:18.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378679 2024-09-15T22:26:18.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378679' 2024-09-15T22:26:18.091 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 81604378679 2024-09-15T22:26:18.092 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:26:18.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378679 -lt 81604378679 2024-09-15T22:26:18.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:26:18.547 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-377957122061 2024-09-15T22:26:18.548 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:26:18.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:26:18.550 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-377957122061 2024-09-15T22:26:18.550 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:26:18.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122061 2024-09-15T22:26:18.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 377957122061' 2024-09-15T22:26:18.552 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 377957122061 2024-09-15T22:26:18.552 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:26:18.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122062 -lt 377957122061 2024-09-15T22:26:18.984 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:26:18.985 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:26:18.985 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:26:19.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:26:19.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:26:19.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:26:19.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:26:19.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:26:19.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:26:19.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:26:19.555 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:26:19.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:26:19.956 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:26:19.956 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:26:19.956 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:26:20.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:26:20.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:26:20.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:26:20.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:26:20.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:26:21.288 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:26:21.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-15T22:26:21.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-0-256 fail 2024-09-15T22:26:21.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:26:21.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:26:21.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-0-256 2024-09-15T22:26:21.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T22:26:21.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T22:26:21.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81867-0-256 td/test-erasure-eio/COPY 2024-09-15T22:26:21.489 INFO:tasks.workunit.client.0.smithi104.stderr:error getting pool-jerasure/obj-size-81867-0-256: (5) Input/output error 2024-09-15T22:26:21.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T22:26:21.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T22:26:21.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:332: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:26:21.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:26:21.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:26:22.021 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:26:22.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:26:22.715 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:26:22.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:26:22.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:26:22.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:26:22.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:26:22.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:26:22.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:26:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:26:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:26:22.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:26:23.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:26:23.051 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:26:23.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:26:23.053 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:26:23.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:26:23.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:26:23.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:26:23.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:26:23.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:26:23.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:26:23.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:26:23.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:26:23.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:26:23.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:26:23.097 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:26:23.097 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:26:23.097 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:26:23.097 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:26:23.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:26:23.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:26:23.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:26:23.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:26:23.102 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:26:23.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:26:23.103 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:26:23.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:26:23.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:26:23.104 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:26:23.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:26:23.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:26:23.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:26:23.106 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:26:23.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:26:23.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:26:23.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:26:23.110 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:26:23.110 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:26:23.110 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:26:23.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:26:23.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:26:23.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:26:23.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T22:26:23.113 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:26:23.113 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:26:23.113 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:26:23.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T22:26:23.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:26:23.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:26:23.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:26:23.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T22:26:23.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T22:26:23.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T22:26:23.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:26:23.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:26:23.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:26:23.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T22:26:23.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T22:26:23.589 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:26:23.589 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:26:23.589 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:26:23.590 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:26:23.590 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:26:23.590 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:26:23.590 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:26:23.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T22:26:23.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:26:23.782 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:26:23.782 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:26:23.782 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:26:23.782 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:26:23.783 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:26:23.783 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:26:23.783 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:26:23.783 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:26:23.784 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:26:23.784 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:26:23.784 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:26:23.784 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:26:23.784 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:26:23.785 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T22:26:23.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:26:23.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:26:23.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:26:23.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:26:23.900 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:26:23.901 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:26:23.901 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:26:23.901 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:26:23.901 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:26:23.901 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:26:23.901 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:26:23.901 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:26:23.901 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:26:23.902 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T22:26:24.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T22:26:24.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T22:26:24.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:26:24.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:26:24.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:26:24.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T22:26:24.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:26:24.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:26:24.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:26:24.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:26:24.941 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:26:24.941 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:26:24.941 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:26:24.941 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:26:24.942 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:26:24.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:26:24.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T22:26:24.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T22:26:25.269 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T22:26:25.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:26:26.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T22:26:26.288 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:26:26.288 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:26:26.288 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:26:26.288 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:26:26.288 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:26:26.289 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:26:26.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T22:26:26.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T22:26:26.389 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:26:26.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T22:26:26.404 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:26:23.635+0000 7ff77d3f2e00 0 load: jerasure load: lrc 2024-09-15T22:26:26.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2024-09-15T22:26:26.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:336: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2024-09-15T22:26:26.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:337: TEST_rados_get_bad_size_shard_1: setup_osds 4 2024-09-15T22:26:26.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T22:26:26.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T22:26:26.406 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T22:26:26.408 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T22:26:26.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T22:26:26.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:26:26.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:26:26.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:26:26.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:26:26.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:26:26.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:26:26.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:26:26.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:26:26.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:26:26.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:26:26.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T22:26:26.415 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:26:26.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4a68f0ec-c9ce-42bd-b4b4-ec572117793c 2024-09-15T22:26:26.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4a68f0ec-c9ce-42bd-b4b4-ec572117793c' 2024-09-15T22:26:26.415 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 4a68f0ec-c9ce-42bd-b4b4-ec572117793c 2024-09-15T22:26:26.415 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:26:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQASX+dmIk7sGRAAPlsPEHt/HuMFeuei4ri/Vg== 2024-09-15T22:26:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQASX+dmIk7sGRAAPlsPEHt/HuMFeuei4ri/Vg=="}' 2024-09-15T22:26:26.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4a68f0ec-c9ce-42bd-b4b4-ec572117793c -i td/test-erasure-eio/0/new.json 2024-09-15T22:26:26.729 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:26:26.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T22:26:26.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASX+dmIk7sGRAAPlsPEHt/HuMFeuei4ri/Vg== --osd-uuid 4a68f0ec-c9ce-42bd-b4b4-ec572117793c 2024-09-15T22:26:26.778 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:26.775+0000 7f5daaff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:26:26.806 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:26.803+0000 7f5daaff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:26:26.807 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:26.803+0000 7f5daaff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:26:26.807 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:26.807+0000 7f5daaff57c0 -1 bdev(0x558acff83c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:26:26.807 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:26.807+0000 7f5daaff57c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T22:26:32.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T22:26:32.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:26:32.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:26:32.883 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T22:26:32.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:26:33.473 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T22:26:33.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:26:33.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:26:33.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:26:33.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:26:33.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:26:33.505 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:33.503+0000 7fd84e06b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:26:33.508 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:33.507+0000 7fd84e06b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:26:33.509 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:33.507+0000 7fd84e06b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:26:33.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:26:33.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:26:33.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:26:33.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:26:33.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:26:33.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:33.939 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:26:33.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:26:33.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:33.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:34.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:35.055 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:35.050+0000 7fd84e06b7c0 -1 Falling back to public interface 2024-09-15T22:26:35.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:35.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:35.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:26:35.386 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:26:35.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:35.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:35.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:36.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:36.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:36.832 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:26:36.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:26:36.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:36.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:37.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:37.850 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:26:37.846+0000 7fd84e06b7c0 -1 osd.0 0 log_to_monitors true 2024-09-15T22:26:38.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:38.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:38.261 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:26:38.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:26:38.261 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:38.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:38.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:39.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:39.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:39.947 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:26:39.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:26:39.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:39.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:40.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:41.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:41.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:41.402 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:26:41.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:26:41.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:41.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:41.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:42.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:42.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:42.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:26:42.993 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:26:42.993 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:42.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:43.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:44.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:44.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:44.442 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:26:44.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:26:44.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:44.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:44.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:45.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:45.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:45.940 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:26:45.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:26:45.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:45.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:46.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:47.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:47.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:47.364 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:26:47.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:26:47.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:47.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:47.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:48.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:48.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:48.793 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:26:48.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:26:48.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:48.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:49.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:50.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:50.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:50.234 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:26:50.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:26:50.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:50.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:51.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:52.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:52.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:52.003 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:26:52.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:26:52.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:52.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:53.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:54.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:54.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:54.077 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:26:54.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:26:54.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:54.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:54.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:55.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:55.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:55.865 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:26:55.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:26:55.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:55.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:57.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:26:58.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:26:58.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:26:58.169 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:26:58.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:26:58.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:26:58.170 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:26:58.999 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:00.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:00.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:00.002 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:27:00.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:27:00.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:00.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:27:00.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:01.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:01.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:01.949 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:27:01.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:27:01.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:01.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:27:02.430 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3268485841,v1:127.0.0.1:6803/3268485841] [v2:127.0.0.1:6804/3268485841,v1:127.0.0.1:6805/3268485841] exists,up 4a68f0ec-c9ce-42bd-b4b4-ec572117793c 2024-09-15T22:27:02.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:27:02.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:27:02.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:27:02.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:27:02.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T22:27:02.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:27:02.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:27:02.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:27:02.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:27:02.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:27:02.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:27:02.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:27:02.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:27:02.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:27:02.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:27:02.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:27:02.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:27:02.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:27:02.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:27:02.436 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:27:02.436 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:27:02.436 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:27:02.436 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:27:02.436 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:27:02.436 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:27:02.436 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:27:02.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:27:02.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:27:02.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:27:02.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=568215fd-f32b-4094-92b7-ea3b19086021 2024-09-15T22:27:02.438 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 568215fd-f32b-4094-92b7-ea3b19086021 2024-09-15T22:27:02.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 568215fd-f32b-4094-92b7-ea3b19086021' 2024-09-15T22:27:02.439 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:27:02.457 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA2X+dm1Qg4GxAABssgwbNBAlb1QTDwM1XQNQ== 2024-09-15T22:27:02.457 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA2X+dm1Qg4GxAABssgwbNBAlb1QTDwM1XQNQ=="}' 2024-09-15T22:27:02.457 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 568215fd-f32b-4094-92b7-ea3b19086021 -i td/test-erasure-eio/1/new.json 2024-09-15T22:27:02.935 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:27:02.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T22:27:02.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2X+dm1Qg4GxAABssgwbNBAlb1QTDwM1XQNQ== --osd-uuid 568215fd-f32b-4094-92b7-ea3b19086021 2024-09-15T22:27:02.986 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:02.982+0000 7f6f082ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:02.989 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:02.986+0000 7f6f082ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:02.990 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:02.986+0000 7f6f082ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:02.990 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:02.986+0000 7f6f082ec7c0 -1 bdev(0x55f881a9bc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:27:02.991 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:02.986+0000 7f6f082ec7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T22:27:09.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T22:27:09.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:27:09.555 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T22:27:09.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:27:09.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:27:10.134 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T22:27:10.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:27:10.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:27:10.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:27:10.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:27:10.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:27:10.166 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:10.162+0000 7fee099f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:10.182 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:10.178+0000 7fee099f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:10.183 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:10.182+0000 7fee099f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:10.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:27:10.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:27:10.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:27:10.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:27:10.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:27:10.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:10.611 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:27:10.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:27:10.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:10.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:11.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:11.454 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:11.450+0000 7fee099f97c0 -1 Falling back to public interface 2024-09-15T22:27:12.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:12.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:12.055 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:27:12.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:27:12.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:12.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:12.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:13.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:13.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:13.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:27:13.534 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:27:13.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:13.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:13.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:14.547 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:14.542+0000 7fee099f97c0 -1 osd.1 0 log_to_monitors true 2024-09-15T22:27:14.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:14.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:14.977 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:27:14.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:27:14.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:14.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:15.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:16.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:16.669 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:16.669 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:27:16.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:27:16.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:16.670 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:17.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:18.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:18.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:18.101 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:27:18.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:27:18.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:18.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:18.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:19.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:19.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:19.643 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:27:19.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:27:19.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:19.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:20.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:21.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:21.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:21.306 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:27:21.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:27:21.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:21.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:21.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:22.751 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:22.751 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:22.751 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:27:22.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:27:22.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:22.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:23.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:24.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:24.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:24.349 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:27:24.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:27:24.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:24.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:24.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:25.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:25.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:25.937 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:27:25.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:27:25.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:25.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:26.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:27.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:27.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:27.658 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:27:27.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:27:27.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:27.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:28.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:29.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:29.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:29.289 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:27:29.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:27:29.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:29.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:30.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:31.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:31.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:31.054 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:27:31.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:27:31.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:31.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:31.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:32.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:32.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:32.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:27:32.785 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:27:32.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:32.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:33.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:34.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:34.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:34.555 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:27:34.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:27:34.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:34.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:35.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:36.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:36.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:36.349 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:27:36.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:27:36.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:36.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:37.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:38.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:38.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:38.037 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:27:38.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:27:38.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:38.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:27:38.600 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1464433614,v1:127.0.0.1:6811/1464433614] [v2:127.0.0.1:6812/1464433614,v1:127.0.0.1:6813/1464433614] exists,up 568215fd-f32b-4094-92b7-ea3b19086021 2024-09-15T22:27:38.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:27:38.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:27:38.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:27:38.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:27:38.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:27:38.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:27:38.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:27:38.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:27:38.604 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:27:38.604 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:27:38.604 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:27:38.606 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:27:38.606 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:27:38.606 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:27:38.606 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:27:38.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:27:38.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:27:38.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:27:38.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:27:38.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:27:38.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:27:38.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:27:38.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:27:38.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:27:38.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:27:38.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:27:38.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:27:38.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:27:38.607 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:27:38.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e7d01e95-a214-4f20-b1e3-81d56a7731dc 2024-09-15T22:27:38.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 e7d01e95-a214-4f20-b1e3-81d56a7731dc' 2024-09-15T22:27:38.608 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 e7d01e95-a214-4f20-b1e3-81d56a7731dc 2024-09-15T22:27:38.608 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:27:38.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBaX+dm4NeaJRAAHjepTVqNMwGLvq+lucnquQ== 2024-09-15T22:27:38.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBaX+dm4NeaJRAAHjepTVqNMwGLvq+lucnquQ=="}' 2024-09-15T22:27:38.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e7d01e95-a214-4f20-b1e3-81d56a7731dc -i td/test-erasure-eio/2/new.json 2024-09-15T22:27:39.380 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:27:39.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T22:27:39.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBaX+dm4NeaJRAAHjepTVqNMwGLvq+lucnquQ== --osd-uuid e7d01e95-a214-4f20-b1e3-81d56a7731dc 2024-09-15T22:27:39.437 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:39.434+0000 7f793a37f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:39.476 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:39.474+0000 7f793a37f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:39.478 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:39.474+0000 7f793a37f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:39.478 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:39.474+0000 7f793a37f7c0 -1 bdev(0x560e6edbbc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:27:39.479 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:39.474+0000 7f793a37f7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T22:27:45.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T22:27:45.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:27:45.811 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T22:27:45.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:27:45.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:27:46.399 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:27:46.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:27:46.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:27:46.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:27:46.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:27:46.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:27:46.432 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:46.430+0000 7f55898a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:46.441 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:46.438+0000 7f55898a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:46.442 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:46.438+0000 7f55898a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:27:46.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:27:46.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:27:46.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:27:46.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:27:46.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:27:46.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:46.825 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:27:46.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:27:46.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:46.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:27:47.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:47.987 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:47.982+0000 7f55898a47c0 -1 Falling back to public interface 2024-09-15T22:27:48.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:48.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:48.236 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:27:48.236 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:27:48.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:48.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:27:48.716 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:49.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:49.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:49.719 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:27:49.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:27:49.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:49.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:27:50.167 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:50.742 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:27:50.738+0000 7f55898a47c0 -1 osd.2 0 log_to_monitors true 2024-09-15T22:27:51.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:51.168 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:51.169 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:27:51.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:27:51.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:51.169 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:27:51.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:52.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:52.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:52.908 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:27:52.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:27:52.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:52.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:27:53.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:54.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:54.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:54.361 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:27:54.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:27:54.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:54.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:27:54.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:55.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:55.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:55.797 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:27:55.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:27:55.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:55.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:27:56.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:57.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:57.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:57.342 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:27:57.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:27:57.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:57.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:27:57.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:27:58.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:27:58.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:27:58.891 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:27:58.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:27:58.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:27:58.892 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:27:59.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:00.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:00.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:00.362 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:28:00.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:28:00.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:00.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:28:00.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:01.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:01.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:01.809 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:28:01.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:28:01.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:01.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:28:02.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:03.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:03.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:03.344 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:28:03.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:28:03.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:03.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:28:03.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:04.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:04.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:04.837 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:28:04.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:28:04.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:04.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:28:05.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:06.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:06.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:06.840 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:28:06.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:28:06.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:06.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:28:07.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:08.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:08.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:08.636 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:28:08.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:28:08.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:08.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:28:09.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:10.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:10.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:10.675 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:28:10.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:28:10.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:10.676 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:28:11.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:12.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:12.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:12.468 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:28:12.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:28:12.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:12.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:28:13.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:14.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:14.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:14.192 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:28:14.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:28:14.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:14.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:28:14.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:15.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:15.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:15.699 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:28:15.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:28:15.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:15.700 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:28:16.326 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2259995505,v1:127.0.0.1:6819/2259995505] [v2:127.0.0.1:6820/2259995505,v1:127.0.0.1:6821/2259995505] exists,up e7d01e95-a214-4f20-b1e3-81d56a7731dc 2024-09-15T22:28:16.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:28:16.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:28:16.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:28:16.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:28:16.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T22:28:16.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:28:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:28:16.330 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:28:16.330 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:28:16.330 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:28:16.331 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:28:16.331 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:28:16.331 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:28:16.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:28:16.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:28:16.333 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:28:16.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4dda9323-fb6c-4a9a-beaa-8dad7466827c 2024-09-15T22:28:16.333 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 4dda9323-fb6c-4a9a-beaa-8dad7466827c 2024-09-15T22:28:16.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 4dda9323-fb6c-4a9a-beaa-8dad7466827c' 2024-09-15T22:28:16.334 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:28:16.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCAX+dmH5k7FRAATJ44uv6qVLbEOHbT/PZDyA== 2024-09-15T22:28:16.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCAX+dmH5k7FRAATJ44uv6qVLbEOHbT/PZDyA=="}' 2024-09-15T22:28:16.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4dda9323-fb6c-4a9a-beaa-8dad7466827c -i td/test-erasure-eio/3/new.json 2024-09-15T22:28:16.946 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:28:16.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T22:28:16.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCAX+dmH5k7FRAATJ44uv6qVLbEOHbT/PZDyA== --osd-uuid 4dda9323-fb6c-4a9a-beaa-8dad7466827c 2024-09-15T22:28:16.998 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:28:16.993+0000 7f7c142987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:28:17.000 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:28:16.997+0000 7f7c142987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:28:17.001 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:28:16.997+0000 7f7c142987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:28:17.002 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:28:16.997+0000 7f7c142987c0 -1 bdev(0x5568690a3c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T22:28:17.002 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:28:16.997+0000 7f7c142987c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T22:28:23.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T22:28:23.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:28:23.534 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T22:28:23.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T22:28:23.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:28:24.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T22:28:24.130 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T22:28:24.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:28:24.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:28:24.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:28:24.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:28:24.161 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:28:24.157+0000 7f11f161f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:28:24.182 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:28:24.177+0000 7f11f161f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:28:24.183 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:28:24.177+0000 7f11f161f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:28:24.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T22:28:24.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:28:24.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:28:24.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:28:24.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:28:24.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:24.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:28:24.646 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:28:24.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:24.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:25.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:25.722 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:28:25.717+0000 7f11f161f7c0 -1 Falling back to public interface 2024-09-15T22:28:26.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:26.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:26.083 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:28:26.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:28:26.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:26.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:26.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:27.566 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:27.567 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:27.567 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:28:27.567 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:28:27.568 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:27.568 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:28.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:28.557 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:28:28.553+0000 7f11f161f7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T22:28:29.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:29.004 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:29.005 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:28:29.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:28:29.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:29.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:29.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:30.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:30.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:30.496 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:28:30.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:28:30.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:30.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:31.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:32.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:32.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:32.193 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:28:32.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:28:32.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:32.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:32.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:33.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:33.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:33.830 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:28:33.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:28:33.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:33.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:34.336 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:35.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:35.334 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:35.334 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:28:35.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:28:35.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:35.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:35.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:36.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:36.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:36.765 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:28:36.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:28:36.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:36.766 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:37.201 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:38.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:38.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:38.204 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:28:38.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:28:38.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:38.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:38.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:39.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:39.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:39.728 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:28:39.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:28:39.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:39.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:40.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:41.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:41.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:41.280 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:28:41.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:28:41.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:41.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:41.775 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:42.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:42.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:42.777 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:28:42.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:28:42.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:42.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:43.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:44.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:44.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:44.802 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:28:44.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:28:44.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:44.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:45.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:46.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:46.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:46.616 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:28:46.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:28:46.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:46.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:47.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:48.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:48.705 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:48.706 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:28:48.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:28:48.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:48.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:49.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:50.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:50.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:50.312 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:28:50.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:28:50.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:50.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:50.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:51.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:51.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:51.777 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:28:51.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:28:51.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:51.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:52.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:28:53.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:28:53.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:28:53.963 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:28:53.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:28:53.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:28:53.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:28:54.436 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/523515774,v1:127.0.0.1:6827/523515774] [v2:127.0.0.1:6828/523515774,v1:127.0.0.1:6829/523515774] exists,up 4dda9323-fb6c-4a9a-beaa-8dad7466827c 2024-09-15T22:28:54.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:28:54.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:28:54.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:28:54.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:28:54.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:28:54.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:28:54.438 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:28:54.438 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:28:54.438 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:28:54.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:28:54.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:28:54.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:28:54.535 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:28:54.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:28:54.627 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:26:35.062+0000 7fd84e06b7c0 0 load: jerasure load: lrc 2024-09-15T22:28:54.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:339: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2024-09-15T22:28:54.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:340: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T22:28:54.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:28:54.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:28:54.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T22:28:54.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:28:54.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T22:28:54.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:28:54.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T22:28:55.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:28:55.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:28:56.612 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:28:56.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:28:57.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:28:57.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:28:57.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:28:57.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:28:57.633 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:28:57.634 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:28:57.634 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:28:57.635 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:28:57.635 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:28:57.635 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:28:57.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:28:57.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:28:57.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:28:57.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:28:57.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:28:57.760 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:28:58.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:28:58.383 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:28:58.383 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:28:58.383 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:28:58.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:28:58.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:28:58.384 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:28:58.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803807 2024-09-15T22:28:58.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803807 2024-09-15T22:28:58.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807' 2024-09-15T22:28:58.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:28:58.541 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:28:58.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574868 2024-09-15T22:28:58.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574868 2024-09-15T22:28:58.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574868' 2024-09-15T22:28:58.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:28:58.690 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:28:58.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378640 2024-09-15T22:28:58.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378640 2024-09-15T22:28:58.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574868 2-81604378640' 2024-09-15T22:28:58.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:28:58.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:28:58.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084294 2024-09-15T22:28:58.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084294 2024-09-15T22:28:58.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574868 2-81604378640 3-120259084294' 2024-09-15T22:28:58.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:28:58.998 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803807 2024-09-15T22:28:58.998 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:28:58.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:28:58.999 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803807 2024-09-15T22:28:58.999 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:28:59.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803807 2024-09-15T22:28:59.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-09-15T22:28:59.001 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803807 2024-09-15T22:28:59.001 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:28:59.457 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803805 -lt 25769803807 2024-09-15T22:28:59.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:29:00.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:29:00.460 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:29:00.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803807 -lt 25769803807 2024-09-15T22:29:00.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:00.885 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574868 2024-09-15T22:29:00.885 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:00.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:29:00.888 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574868 2024-09-15T22:29:00.888 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:00.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574868 2024-09-15T22:29:00.890 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574868 2024-09-15T22:29:00.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574868' 2024-09-15T22:29:00.890 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:29:01.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574869 -lt 55834574868 2024-09-15T22:29:01.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:01.364 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378640 2024-09-15T22:29:01.364 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:01.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:29:01.367 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378640 2024-09-15T22:29:01.367 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:01.369 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 81604378640 2024-09-15T22:29:01.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378640 2024-09-15T22:29:01.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378640' 2024-09-15T22:29:01.369 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:29:01.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378640 -lt 81604378640 2024-09-15T22:29:01.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:01.805 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084294 2024-09-15T22:29:01.805 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:01.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:29:01.808 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084294 2024-09-15T22:29:01.808 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:01.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084294 2024-09-15T22:29:01.809 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 120259084294 2024-09-15T22:29:01.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084294' 2024-09-15T22:29:01.810 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:29:02.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084295 -lt 120259084294 2024-09-15T22:29:02.282 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:29:02.282 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:29:02.283 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:29:02.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:29:02.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:29:02.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:29:02.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:29:02.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:29:02.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:29:02.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:29:02.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:29:03.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:29:03.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:29:03.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:29:03.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:29:03.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:29:03.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:342: TEST_rados_get_bad_size_shard_1: local shard_id=1 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:343: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=10 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T22:29:03.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-81867-1-10 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81867-1-10 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81867-1-10 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:29:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81867-1-10 td/test-erasure-eio/ORIGINAL 2024-09-15T22:29:03.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-81867-1-10 td/test-erasure-eio 1 10 set 2024-09-15T22:29:03.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-1-10 2024-09-15T22:29:03.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:29:03.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:29:03.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:29:03.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-15T22:29:03.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:29:03.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-15T22:29:03.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:29:03.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-15T22:29:03.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:29:03.962 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-1-10 2024-09-15T22:29:03.962 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:29:03.962 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-1-10 2024-09-15T22:29:03.963 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:29:03.963 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-1-10 2024-09-15T22:29:04.378 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:29:04.379 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:29:04.379 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:29:04.379 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:29:04.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:29:04.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:29:04.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-15T22:29:04.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:29:05.035 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:29:05.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-15T22:29:05.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-15T22:29:05.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-15T22:29:05.058 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records in 2024-09-15T22:29:05.058 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records out 2024-09-15T22:29:05.058 INFO:tasks.workunit.client.0.smithi104.stderr:10 bytes copied, 0.000195451 s, 51.2 kB/s 2024-09-15T22:29:05.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81867-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:05.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:29:05.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:29:05.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T22:29:05.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:29:05.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81867-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:05.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:29:05.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:29:05.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T22:29:05.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:29:05.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T22:29:05.061 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:29:05.061 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:29:05.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:29:05.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:29:05.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:29:06.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:29:06.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81867-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:06.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:29:06.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:29:06.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T22:29:06.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:29:06.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T22:29:06.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81867-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:08.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T22:29:08.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:29:08.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:29:08.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T22:29:08.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:29:08.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:29:08.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:29:08.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:29:08.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:29:08.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:29:08.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:29:08.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:29:08.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:29:08.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:29:08.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:29:08.616 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:29:08.616 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:29:08.616 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:29:08.617 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:29:08.617 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:29:08.617 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:29:08.617 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:29:08.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:29:08.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:29:08.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:29:08.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:29:08.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:29:08.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:29:08.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:29:08.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:29:08.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:29:08.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:29:08.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:29:08.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T22:29:08.619 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.1 2024-09-15T22:29:08.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:29:08.619 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T22:29:08.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T22:29:08.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:29:08.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:29:08.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:29:08.653 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:08.649+0000 7f9682c6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:08.656 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:08.653+0000 7f9682c6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:08.657 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:08.653+0000 7f9682c6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:09.100 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T22:29:09.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:29:09.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:29:09.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:29:09.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:29:09.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:09.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:29:09.101 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:29:09.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:09.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:29:09.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:29:09.706 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:09.701+0000 7f9682c6a7c0 -1 Falling back to public interface 2024-09-15T22:29:10.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:29:10.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:10.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:29:10.536 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:29:10.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:10.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:29:10.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:29:11.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:29:11.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:11.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:29:11.961 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:29:11.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:11.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:29:12.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:29:12.732 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:12.729+0000 7f9682c6a7c0 -1 osd.1 40 log_to_monitors true 2024-09-15T22:29:13.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:29:13.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:13.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:29:13.430 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:29:13.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:13.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:29:14.116 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 41 last_clean_interval [13,40) [v2:127.0.0.1:6810/3171969363,v1:127.0.0.1:6811/3171969363] [v2:127.0.0.1:6812/3171969363,v1:127.0.0.1:6813/3171969363] exists,up 568215fd-f32b-4094-92b7-ea3b19086021 2024-09-15T22:29:14.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:29:14.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:29:14.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:29:14.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:29:14.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:29:14.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:29:14.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:29:14.117 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:29:14.118 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:29:14.119 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:29:14.119 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:29:14.119 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:29:14.119 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:29:14.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:29:14.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:29:14.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:29:14.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:29:14.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:29:14.238 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:29:14.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:29:14.863 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:29:14.863 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:29:14.863 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:29:14.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:29:14.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:14.863 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:29:15.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803812 2024-09-15T22:29:15.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803812 2024-09-15T22:29:15.058 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812' 2024-09-15T22:29:15.058 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:15.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:29:15.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528323 2024-09-15T22:29:15.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528323 2024-09-15T22:29:15.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-193273528323' 2024-09-15T22:29:15.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:15.215 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:29:15.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378646 2024-09-15T22:29:15.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378646 2024-09-15T22:29:15.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-193273528323 2-81604378646' 2024-09-15T22:29:15.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:15.375 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:29:15.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084300 2024-09-15T22:29:15.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084300 2024-09-15T22:29:15.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-193273528323 2-81604378646 3-120259084300' 2024-09-15T22:29:15.528 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:15.529 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:15.529 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803812 2024-09-15T22:29:15.531 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:29:15.531 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:15.531 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803812 2024-09-15T22:29:15.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803812 2024-09-15T22:29:15.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-15T22:29:15.533 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803812 2024-09-15T22:29:15.533 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:29:15.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-15T22:29:15.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:15.980 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-193273528323 2024-09-15T22:29:15.980 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:15.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:29:15.983 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-193273528323 2024-09-15T22:29:15.983 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:15.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528323 2024-09-15T22:29:15.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 193273528323' 2024-09-15T22:29:15.985 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 193273528323 2024-09-15T22:29:15.985 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:29:16.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528323 -lt 193273528323 2024-09-15T22:29:16.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:16.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:16.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378646 2024-09-15T22:29:16.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:29:16.416 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378646 2024-09-15T22:29:16.416 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:16.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378646 2024-09-15T22:29:16.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378646' 2024-09-15T22:29:16.417 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 81604378646 2024-09-15T22:29:16.417 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:29:16.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378646 -lt 81604378646 2024-09-15T22:29:16.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:16.885 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:16.885 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084300 2024-09-15T22:29:16.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:29:16.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084300 2024-09-15T22:29:16.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:16.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084300 2024-09-15T22:29:16.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084300' 2024-09-15T22:29:16.888 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 120259084300 2024-09-15T22:29:16.889 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:29:17.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084300 -lt 120259084300 2024-09-15T22:29:17.327 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:29:17.327 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:29:17.328 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:29:18.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:29:18.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:29:18.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:29:18.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:29:18.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:29:18.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:29:18.012 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:29:18.012 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:29:18.423 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:29:18.424 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:29:18.424 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:29:18.424 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:29:18.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:29:18.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:29:18.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:29:18.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:29:18.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:29:19.720 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:29:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-1-10 2024-09-15T22:29:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:29:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:29:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-1-10 2024-09-15T22:29:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:29:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:29:19.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81867-1-10 td/test-erasure-eio/COPY 2024-09-15T22:29:19.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:29:19.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:29:19.850 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2024-09-15T22:29:19.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2024-09-15T22:29:19.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-81867-1-10 td/test-erasure-eio 2 10 set 2024-09-15T22:29:19.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-1-10 2024-09-15T22:29:19.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:29:19.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:29:19.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:29:19.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2024-09-15T22:29:19.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:29:19.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-15T22:29:19.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:29:19.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-15T22:29:19.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:29:19.853 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-1-10 2024-09-15T22:29:19.853 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:29:19.853 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-1-10 2024-09-15T22:29:19.854 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-1-10 2024-09-15T22:29:19.854 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:29:20.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:29:20.360 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:29:20.360 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:29:20.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:29:20.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:29:20.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:29:20.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2024-09-15T22:29:20.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:29:21.017 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:29:21.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-15T22:29:21.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-15T22:29:21.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-15T22:29:21.039 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records in 2024-09-15T22:29:21.039 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records out 2024-09-15T22:29:21.039 INFO:tasks.workunit.client.0.smithi104.stderr:10 bytes copied, 0.000195464 s, 51.2 kB/s 2024-09-15T22:29:21.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-81867-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:21.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:29:21.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:29:21.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T22:29:21.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:29:21.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-81867-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:21.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:29:21.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:29:21.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T22:29:21.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:29:21.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T22:29:21.041 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:29:21.041 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:29:21.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:29:21.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:29:21.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:29:21.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:29:21.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-81867-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:21.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:29:21.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:29:21.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T22:29:21.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:29:21.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-15T22:29:21.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-81867-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:24.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-15T22:29:24.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:29:24.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:29:24.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T22:29:24.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:29:24.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:29:24.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:29:24.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:29:24.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:29:24.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:29:24.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:29:24.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:29:24.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:29:24.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:29:24.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:29:24.012 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:29:24.012 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:29:24.012 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:29:24.014 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:29:24.014 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:29:24.014 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:29:24.014 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:29:24.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:29:24.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:29:24.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:29:24.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:29:24.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:29:24.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:29:24.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:29:24.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:29:24.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:29:24.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:29:24.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:29:24.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T22:29:24.015 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.2 2024-09-15T22:29:24.015 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:29:24.015 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T22:29:24.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T22:29:24.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:29:24.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:29:24.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:29:24.048 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:24.045+0000 7fd6c2a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:24.051 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:24.049+0000 7fd6c2a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:24.053 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:24.049+0000 7fd6c2a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:24.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T22:29:24.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:29:24.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:29:24.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:29:24.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:29:24.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:24.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:29:24.515 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:29:24.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:24.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:29:24.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:29:25.310 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:25.304+0000 7fd6c2a897c0 -1 Falling back to public interface 2024-09-15T22:29:25.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:29:25.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:25.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:29:25.957 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:29:25.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:25.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:29:26.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:29:27.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:29:27.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:27.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:29:27.468 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:29:27.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:27.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:29:27.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:29:28.089 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:28.084+0000 7fd6c2a897c0 -1 osd.2 49 log_to_monitors true 2024-09-15T22:29:28.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:29:28.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:28.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:29:28.953 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:29:28.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:28.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:29:29.587 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 up in weight 1 up_from 56 up_thru 41 down_at 52 last_clean_interval [19,49) [v2:127.0.0.1:6818/2825763299,v1:127.0.0.1:6819/2825763299] [v2:127.0.0.1:6820/2825763299,v1:127.0.0.1:6821/2825763299] exists,up e7d01e95-a214-4f20-b1e3-81d56a7731dc 2024-09-15T22:29:29.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:29:29.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:29:29.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:29:29.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:29:29.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:29:29.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:29:29.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:29:29.588 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:29:29.588 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:29:29.588 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:29:29.589 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:29:29.589 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:29:29.589 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:29:29.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:29:29.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:29:29.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:29:29.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:29:29.706 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:29:29.706 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:29:30.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:29:30.361 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:29:30.361 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:29:30.361 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:29:30.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:29:30.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:30.361 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:29:30.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803817 2024-09-15T22:29:30.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803817 2024-09-15T22:29:30.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803817' 2024-09-15T22:29:30.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:30.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:29:30.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528328 2024-09-15T22:29:30.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528328 2024-09-15T22:29:30.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803817 1-193273528328' 2024-09-15T22:29:30.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:30.676 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:29:30.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168579 2024-09-15T22:29:30.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168579 2024-09-15T22:29:30.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803817 1-193273528328 2-240518168579' 2024-09-15T22:29:30.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:30.828 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:29:30.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084305 2024-09-15T22:29:30.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084305 2024-09-15T22:29:30.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803817 1-193273528328 2-240518168579 3-120259084305' 2024-09-15T22:29:30.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:30.984 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803817 2024-09-15T22:29:30.984 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:30.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:29:30.986 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803817 2024-09-15T22:29:30.987 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:30.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803817 2024-09-15T22:29:30.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2024-09-15T22:29:30.988 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803817 2024-09-15T22:29:30.988 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:29:31.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803815 -lt 25769803817 2024-09-15T22:29:31.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:29:32.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:29:32.410 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:29:32.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803818 -lt 25769803817 2024-09-15T22:29:32.878 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:32.878 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-193273528328 2024-09-15T22:29:32.878 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:32.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:29:32.880 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-193273528328 2024-09-15T22:29:32.880 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:32.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528328 2024-09-15T22:29:32.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 193273528328' 2024-09-15T22:29:32.882 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 193273528328 2024-09-15T22:29:32.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:29:33.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528328 -lt 193273528328 2024-09-15T22:29:33.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:33.307 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-240518168579 2024-09-15T22:29:33.308 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:33.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:29:33.311 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-240518168579 2024-09-15T22:29:33.311 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:33.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168579 2024-09-15T22:29:33.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 240518168579' 2024-09-15T22:29:33.313 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 240518168579 2024-09-15T22:29:33.313 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:29:33.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168579 -lt 240518168579 2024-09-15T22:29:33.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:33.764 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:33.764 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084305 2024-09-15T22:29:33.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:29:33.765 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084305 2024-09-15T22:29:33.766 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:33.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084305 2024-09-15T22:29:33.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084305' 2024-09-15T22:29:33.768 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 120259084305 2024-09-15T22:29:33.768 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:29:34.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084305 -lt 120259084305 2024-09-15T22:29:34.215 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:29:34.216 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:29:34.216 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:29:34.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:29:34.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:29:34.790 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:29:34.790 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:29:34.790 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:29:34.790 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:29:34.791 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:29:34.791 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:29:35.188 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:29:35.189 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:29:35.189 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:29:35.189 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:29:35.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:29:35.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:29:35.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:29:35.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:29:35.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:29:36.433 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:29:36.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-15T22:29:36.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-1-10 2024-09-15T22:29:36.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:29:36.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:29:36.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-1-10 2024-09-15T22:29:36.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:29:36.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:29:36.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81867-1-10 td/test-erasure-eio/COPY 2024-09-15T22:29:36.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:29:36.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:29:36.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T22:29:36.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:344: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 0 2024-09-15T22:29:36.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T22:29:36.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-15T22:29:36.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2024-09-15T22:29:36.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-15T22:29:36.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=0 2024-09-15T22:29:36.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-15T22:29:36.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-81867-1-0 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81867-1-0 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81867-1-0 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:29:36.613 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:29:36.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81867-1-0 td/test-erasure-eio/ORIGINAL 2024-09-15T22:29:36.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-81867-1-0 td/test-erasure-eio 1 0 set 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-1-0 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-1-0 2024-09-15T22:29:36.799 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:29:36.800 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-1-0 2024-09-15T22:29:36.800 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:29:36.800 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-1-0 2024-09-15T22:29:37.259 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:29:37.259 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:29:37.259 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:29:37.259 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:29:37.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:29:37.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:29:37.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-15T22:29:37.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:29:37.939 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:29:37.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-15T22:29:37.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-15T22:29:37.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-15T22:29:37.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81867-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:37.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:29:37.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:29:37.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T22:29:37.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:29:37.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81867-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:37.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:29:37.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:29:37.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T22:29:37.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:29:37.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T22:29:37.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:29:37.964 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:29:37.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:29:37.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:29:37.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:29:39.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:29:39.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81867-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:39.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:29:39.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:29:39.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T22:29:39.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:29:39.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T22:29:39.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81867-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:41.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:29:41.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:29:41.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:29:41.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:29:41.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:29:41.352 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:29:41.353 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:29:41.353 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:29:41.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:29:41.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T22:29:41.355 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.1 2024-09-15T22:29:41.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:29:41.355 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T22:29:41.356 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T22:29:41.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:29:41.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:29:41.357 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:29:41.394 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:41.388+0000 7f543cc5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:41.397 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:41.392+0000 7f543cc5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:41.399 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:41.392+0000 7f543cc5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:41.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T22:29:41.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:29:41.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:29:41.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:29:41.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:29:41.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:41.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:29:41.861 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:29:41.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:41.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:29:42.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:29:42.470 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:42.464+0000 7f543cc5c7c0 -1 Falling back to public interface 2024-09-15T22:29:43.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:29:43.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:43.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:29:43.295 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:29:43.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:43.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:29:43.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:29:44.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:29:44.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:44.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:29:44.771 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:29:44.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:44.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:29:45.127 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:45.120+0000 7f543cc5c7c0 -1 osd.1 61 log_to_monitors true 2024-09-15T22:29:45.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:29:46.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:29:46.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:46.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:29:46.217 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:29:46.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:46.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:29:46.902 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [45,61) [v2:127.0.0.1:6810/2437121980,v1:127.0.0.1:6811/2437121980] [v2:127.0.0.1:6812/2437121980,v1:127.0.0.1:6813/2437121980] exists,up 568215fd-f32b-4094-92b7-ea3b19086021 2024-09-15T22:29:46.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:29:46.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:29:46.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:29:46.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:29:46.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:29:46.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:29:46.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:29:46.903 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:29:46.904 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:29:46.904 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:29:46.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:29:46.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:29:46.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:29:47.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:29:47.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:29:47.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:29:47.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:29:47.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:29:47.023 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:29:47.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:29:47.530 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:29:47.530 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:29:47.530 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:29:47.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:29:47.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:47.531 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:29:47.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803823 2024-09-15T22:29:47.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803823 2024-09-15T22:29:47.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803823' 2024-09-15T22:29:47.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:47.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:29:47.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808835 2024-09-15T22:29:47.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808835 2024-09-15T22:29:47.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803823 1-287762808835' 2024-09-15T22:29:47.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:47.837 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:29:47.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168584 2024-09-15T22:29:47.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168584 2024-09-15T22:29:47.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803823 1-287762808835 2-240518168584' 2024-09-15T22:29:47.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:29:47.995 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:29:48.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084310 2024-09-15T22:29:48.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084310 2024-09-15T22:29:48.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803823 1-287762808835 2-240518168584 3-120259084310' 2024-09-15T22:29:48.152 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:48.152 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803823 2024-09-15T22:29:48.152 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:48.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:29:48.156 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803823 2024-09-15T22:29:48.156 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:48.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803823 2024-09-15T22:29:48.157 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803823' 2024-09-15T22:29:48.157 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803823 2024-09-15T22:29:48.158 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:29:48.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803823 -lt 25769803823 2024-09-15T22:29:48.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:48.624 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:48.624 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-287762808835 2024-09-15T22:29:48.625 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:29:48.626 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-287762808835 2024-09-15T22:29:48.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:48.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808835 2024-09-15T22:29:48.628 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 287762808835' 2024-09-15T22:29:48.628 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 287762808835 2024-09-15T22:29:48.629 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:29:49.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 287762808835 2024-09-15T22:29:49.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:29:50.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:29:50.111 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:29:50.537 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808835 -lt 287762808835 2024-09-15T22:29:50.537 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:50.538 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-240518168584 2024-09-15T22:29:50.538 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:50.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:29:50.541 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-240518168584 2024-09-15T22:29:50.541 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:50.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168584 2024-09-15T22:29:50.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 240518168584' 2024-09-15T22:29:50.542 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 240518168584 2024-09-15T22:29:50.542 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:29:51.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168585 -lt 240518168584 2024-09-15T22:29:51.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:29:51.015 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084310 2024-09-15T22:29:51.015 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:29:51.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:29:51.017 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084310 2024-09-15T22:29:51.018 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:29:51.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084310 2024-09-15T22:29:51.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084310' 2024-09-15T22:29:51.020 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 120259084310 2024-09-15T22:29:51.021 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:29:51.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084311 -lt 120259084310 2024-09-15T22:29:51.474 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:29:51.475 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:29:51.475 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:29:52.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:29:52.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:29:52.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:29:52.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:29:52.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:29:52.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:29:52.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:29:52.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:29:52.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:29:52.444 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:29:52.444 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:29:52.445 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:29:52.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:29:52.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:29:52.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:29:52.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:29:52.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:29:53.700 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:29:53.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-1-0 2024-09-15T22:29:53.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:29:53.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:29:53.719 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-1-0 2024-09-15T22:29:53.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:29:53.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:29:53.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81867-1-0 td/test-erasure-eio/COPY 2024-09-15T22:29:53.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:29:53.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:29:53.878 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2024-09-15T22:29:53.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2024-09-15T22:29:53.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-81867-1-0 td/test-erasure-eio 2 0 set 2024-09-15T22:29:53.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-1-0 2024-09-15T22:29:53.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:29:53.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:29:53.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:29:53.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2024-09-15T22:29:53.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:29:53.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-15T22:29:53.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:29:53.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-15T22:29:53.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:29:53.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-1-0 2024-09-15T22:29:53.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:29:53.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-1-0 2024-09-15T22:29:53.883 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-1-0 2024-09-15T22:29:53.883 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:29:54.341 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:29:54.341 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:29:54.341 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:29:54.341 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:29:54.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:29:54.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:29:54.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2024-09-15T22:29:54.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:29:55.013 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:29:55.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-15T22:29:55.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-15T22:29:55.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-15T22:29:55.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-81867-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:55.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:29:55.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:29:55.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T22:29:55.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:29:55.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-81867-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:55.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:29:55.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:29:55.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T22:29:55.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:29:55.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T22:29:55.041 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:29:55.041 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:29:55.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:29:55.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:29:55.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:29:56.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:29:56.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-81867-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:56.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:29:56.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:29:56.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T22:29:56.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:29:56.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-15T22:29:56.355 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-81867-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:29:58.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-15T22:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T22:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:29:58.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:29:58.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:29:58.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:29:58.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:29:58.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:29:58.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:29:58.297 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:29:58.297 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:29:58.297 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:29:58.299 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:29:58.300 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:29:58.300 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:29:58.300 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:29:58.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:29:58.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:29:58.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:29:58.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:29:58.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:29:58.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:29:58.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:29:58.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:29:58.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:29:58.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:29:58.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:29:58.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T22:29:58.301 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.2 2024-09-15T22:29:58.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:29:58.302 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T22:29:58.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T22:29:58.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:29:58.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:29:58.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:29:58.334 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:58.328+0000 7f46ed9ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:58.337 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:58.332+0000 7f46ed9ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:58.339 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:58.332+0000 7f46ed9ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:29:58.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T22:29:58.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:29:58.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:29:58.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:29:58.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:29:58.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:29:58.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:29:58.778 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:29:58.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:29:58.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:29:59.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:29:59.586 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:29:59.580+0000 7f46ed9ca7c0 -1 Falling back to public interface 2024-09-15T22:30:00.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:00.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:00.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:30:00.287 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:30:00.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:00.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:30:00.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:01.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:01.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:01.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:30:01.796 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:30:01.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:01.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:30:02.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:02.478 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:02.472+0000 7f46ed9ca7c0 -1 osd.2 73 log_to_monitors true 2024-09-15T22:30:03.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:30:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:30:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:03.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:30:03.784 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 up in weight 1 up_from 78 up_thru 63 down_at 74 last_clean_interval [56,73) [v2:127.0.0.1:6818/3759675987,v1:127.0.0.1:6819/3759675987] [v2:127.0.0.1:6820/3759675987,v1:127.0.0.1:6821/3759675987] exists,up e7d01e95-a214-4f20-b1e3-81d56a7731dc 2024-09-15T22:30:03.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:30:03.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:30:03.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:30:03.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:30:03.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:30:03.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:30:03.785 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:30:03.786 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:30:03.786 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:30:03.786 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:30:03.786 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:30:03.786 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:30:03.786 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:30:03.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:30:03.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:30:03.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:30:03.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:30:03.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:30:03.891 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:30:04.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:30:04.387 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:30:04.387 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:30:04.387 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:30:04.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:30:04.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:04.388 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:30:04.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803828 2024-09-15T22:30:04.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803828 2024-09-15T22:30:04.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803828' 2024-09-15T22:30:04.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:04.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:30:04.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808840 2024-09-15T22:30:04.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808840 2024-09-15T22:30:04.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803828 1-287762808840' 2024-09-15T22:30:04.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:04.786 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:30:04.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449091 2024-09-15T22:30:04.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449091 2024-09-15T22:30:04.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803828 1-287762808840 2-335007449091' 2024-09-15T22:30:04.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:04.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:30:05.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084316 2024-09-15T22:30:05.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084316 2024-09-15T22:30:05.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803828 1-287762808840 2-335007449091 3-120259084316' 2024-09-15T22:30:05.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:05.104 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803828 2024-09-15T22:30:05.104 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:05.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:30:05.105 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803828 2024-09-15T22:30:05.106 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:05.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803828 2024-09-15T22:30:05.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803828' 2024-09-15T22:30:05.107 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803828 2024-09-15T22:30:05.107 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:30:05.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803826 -lt 25769803828 2024-09-15T22:30:05.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:30:06.568 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:30:06.568 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:30:06.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803828 -lt 25769803828 2024-09-15T22:30:06.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:06.987 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-287762808840 2024-09-15T22:30:06.988 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:06.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:30:06.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-287762808840 2024-09-15T22:30:06.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:06.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808840 2024-09-15T22:30:06.991 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 287762808840' 2024-09-15T22:30:06.991 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 287762808840 2024-09-15T22:30:06.992 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:30:07.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808841 -lt 287762808840 2024-09-15T22:30:07.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:07.410 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449091 2024-09-15T22:30:07.410 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:07.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:30:07.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449091 2024-09-15T22:30:07.413 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:07.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449091 2024-09-15T22:30:07.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449091' 2024-09-15T22:30:07.415 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 335007449091 2024-09-15T22:30:07.415 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:30:07.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449091 -lt 335007449091 2024-09-15T22:30:07.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:07.928 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:07.928 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084316 2024-09-15T22:30:07.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:30:07.930 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084316 2024-09-15T22:30:07.931 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:07.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084316 2024-09-15T22:30:07.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084316' 2024-09-15T22:30:07.932 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 120259084316 2024-09-15T22:30:07.933 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:30:08.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084316 -lt 120259084316 2024-09-15T22:30:08.380 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:30:08.381 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:30:08.381 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:30:08.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:30:08.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:30:08.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:30:08.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:30:08.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:30:08.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:30:08.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:30:08.934 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:30:09.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:30:09.359 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:30:09.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:30:09.360 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:30:09.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:30:09.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:30:09.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:30:09.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:30:09.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:30:10.673 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:30:10.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-15T22:30:10.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-1-0 2024-09-15T22:30:10.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:30:10.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:30:10.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-1-0 2024-09-15T22:30:10.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:30:10.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:30:10.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81867-1-0 td/test-erasure-eio/COPY 2024-09-15T22:30:10.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:30:10.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:30:10.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T22:30:10.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:345: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 256 add 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=256 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=add 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-81867-1-256 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81867-1-256 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:30:10.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:30:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81867-1-256 2024-09-15T22:30:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:30:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:30:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:30:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:30:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:30:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:30:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:30:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:30:10.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81867-1-256 td/test-erasure-eio/ORIGINAL 2024-09-15T22:30:11.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-81867-1-256 td/test-erasure-eio 1 256 add 2024-09-15T22:30:11.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-1-256 2024-09-15T22:30:11.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:30:11.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:30:11.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:30:11.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-15T22:30:11.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:30:11.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-15T22:30:11.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:30:11.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-15T22:30:11.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:30:11.073 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-1-256 2024-09-15T22:30:11.073 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:30:11.073 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-1-256 2024-09-15T22:30:11.073 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-1-256 2024-09-15T22:30:11.073 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:30:11.540 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:30:11.541 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:30:11.541 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:30:11.541 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:30:11.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:30:11.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:30:11.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-15T22:30:11.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:30:12.413 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:30:12.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-15T22:30:12.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81867-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:12.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:30:12.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:30:12.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T22:30:12.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:30:12.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81867-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:12.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:30:12.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:30:12.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T22:30:12.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:30:12.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T22:30:12.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:30:12.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:30:12.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:30:12.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:30:12.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:30:12.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:30:12.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81867-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:12.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:30:12.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:30:12.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T22:30:12.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:30:12.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T22:30:12.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81867-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:13.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T22:30:13.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:30:13.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:30:13.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T22:30:13.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:30:13.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:30:13.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:30:13.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:30:13.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:30:13.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:30:13.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:30:13.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:30:13.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:30:13.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:30:13.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:30:13.541 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:30:13.541 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:30:13.541 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:30:13.543 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:30:13.543 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:30:13.543 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:30:13.543 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:30:13.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:30:13.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.1 2024-09-15T22:30:13.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:30:13.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T22:30:13.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T22:30:13.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:30:13.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:30:13.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:30:13.584 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:13.580+0000 7f6ba17617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:13.638 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:13.632+0000 7f6ba17617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:13.639 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:13.632+0000 7f6ba17617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:14.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T22:30:14.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:30:14.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:30:14.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:30:14.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:30:14.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:14.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:30:14.013 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:30:14.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:14.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:30:14.500 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:14.890 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:14.884+0000 7f6ba17617c0 -1 Falling back to public interface 2024-09-15T22:30:15.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:15.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:15.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:30:15.502 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:30:15.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:15.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:30:15.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:16.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:16.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:16.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:30:16.981 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:30:16.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:16.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:30:17.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:18.183 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:18.176+0000 7f6ba17617c0 -1 osd.1 84 log_to_monitors true 2024-09-15T22:30:18.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:18.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:18.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:30:18.403 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:30:18.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:18.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:30:19.329 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 up in weight 1 up_from 89 up_thru 78 down_at 85 last_clean_interval [67,84) [v2:127.0.0.1:6810/1653108267,v1:127.0.0.1:6811/1653108267] [v2:127.0.0.1:6812/1653108267,v1:127.0.0.1:6813/1653108267] exists,up 568215fd-f32b-4094-92b7-ea3b19086021 2024-09-15T22:30:19.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:30:19.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:30:19.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:30:19.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:30:19.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:30:19.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:30:19.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:30:19.330 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:30:19.330 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:30:19.330 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:30:19.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:30:19.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:30:19.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:30:19.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:30:19.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:30:19.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:30:19.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:30:19.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:30:19.448 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:30:19.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:30:19.979 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:30:19.979 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:30:19.979 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:30:19.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:30:19.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:19.979 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:30:20.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803833 2024-09-15T22:30:20.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803833 2024-09-15T22:30:20.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803833' 2024-09-15T22:30:20.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:20.245 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:30:20.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089347 2024-09-15T22:30:20.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089347 2024-09-15T22:30:20.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803833 1-382252089347' 2024-09-15T22:30:20.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:20.403 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:30:20.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449096 2024-09-15T22:30:20.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449096 2024-09-15T22:30:20.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803833 1-382252089347 2-335007449096' 2024-09-15T22:30:20.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:20.598 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:30:20.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084321 2024-09-15T22:30:20.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084321 2024-09-15T22:30:20.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803833 1-382252089347 2-335007449096 3-120259084321' 2024-09-15T22:30:20.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:20.756 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803833 2024-09-15T22:30:20.756 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:20.757 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:30:20.758 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803833 2024-09-15T22:30:20.758 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:20.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803833 2024-09-15T22:30:20.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803833' 2024-09-15T22:30:20.759 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803833 2024-09-15T22:30:20.760 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:30:21.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803831 -lt 25769803833 2024-09-15T22:30:21.242 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:30:22.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:30:22.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:30:22.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803834 -lt 25769803833 2024-09-15T22:30:22.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:22.727 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-382252089347 2024-09-15T22:30:22.727 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:22.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:30:22.729 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-382252089347 2024-09-15T22:30:22.730 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:22.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089347 2024-09-15T22:30:22.731 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 382252089347' 2024-09-15T22:30:22.731 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 382252089347 2024-09-15T22:30:22.731 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:30:23.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 382252089347 -lt 382252089347 2024-09-15T22:30:23.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:23.174 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449096 2024-09-15T22:30:23.175 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:23.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:30:23.177 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449096 2024-09-15T22:30:23.177 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:23.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449096 2024-09-15T22:30:23.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449096' 2024-09-15T22:30:23.179 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 335007449096 2024-09-15T22:30:23.179 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:30:23.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449096 -lt 335007449096 2024-09-15T22:30:23.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:23.662 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084321 2024-09-15T22:30:23.662 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:23.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:30:23.665 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084321 2024-09-15T22:30:23.665 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:23.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084321 2024-09-15T22:30:23.666 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084321' 2024-09-15T22:30:23.666 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 120259084321 2024-09-15T22:30:23.667 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:30:24.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084321 -lt 120259084321 2024-09-15T22:30:24.122 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:30:24.122 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:30:24.122 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:30:24.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:30:24.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:30:24.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:30:24.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:30:24.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:30:24.681 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:30:24.682 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:30:24.682 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:30:25.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:30:25.109 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:30:25.109 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:30:25.109 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:30:25.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:30:25.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:30:25.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:30:25.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-15T22:30:25.648 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records in 2024-09-15T22:30:25.648 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records out 2024-09-15T22:30:25.648 INFO:tasks.workunit.client.0.smithi104.stderr:256 bytes copied, 4.5097e-05 s, 5.7 MB/s 2024-09-15T22:30:25.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81867-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81867-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:30:25.650 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:30:25.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:30:25.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:30:25.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:30:25.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:30:25.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81867-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:25.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:30:25.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:30:25.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T22:30:25.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:30:25.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T22:30:25.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81867-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T22:30:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:30:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:30:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T22:30:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:30:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:30:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:30:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:30:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:30:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:30:28.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:30:28.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:30:28.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:30:28.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:30:28.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:30:28.813 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:30:28.813 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:30:28.813 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:30:28.815 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:30:28.815 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:30:28.815 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:30:28.815 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:30:28.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:30:28.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:30:28.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:30:28.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:30:28.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:30:28.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:30:28.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:30:28.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:30:28.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:30:28.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:30:28.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:30:28.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T22:30:28.816 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.1 2024-09-15T22:30:28.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:30:28.817 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T22:30:28.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T22:30:28.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:30:28.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:30:28.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:30:28.851 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:28.844+0000 7f6898c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:28.854 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:28.848+0000 7f6898c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:28.855 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:28.852+0000 7f6898c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:29.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T22:30:29.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:30:29.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:30:29.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:30:29.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:30:29.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:29.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:30:29.305 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:30:29.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:29.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:30:29.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:29.882 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:29.876+0000 7f6898c4b7c0 -1 Falling back to public interface 2024-09-15T22:30:30.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:30.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:30.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:30:30.817 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:30:30.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:30.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:30:31.243 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:32.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:32.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:32.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:30:32.245 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:30:32.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:32.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:30:32.554 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:32.548+0000 7f6898c4b7c0 -1 osd.1 91 log_to_monitors true 2024-09-15T22:30:32.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:33.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:33.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:33.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:30:33.675 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:30:33.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:33.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:30:34.329 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 up in weight 1 up_from 96 up_thru 96 down_at 92 last_clean_interval [89,91) [v2:127.0.0.1:6810/2921058155,v1:127.0.0.1:6811/2921058155] [v2:127.0.0.1:6812/2921058155,v1:127.0.0.1:6813/2921058155] exists,up 568215fd-f32b-4094-92b7-ea3b19086021 2024-09-15T22:30:34.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:30:34.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:30:34.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:30:34.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:30:34.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:30:34.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:30:34.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:30:34.330 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:30:34.330 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:30:34.330 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:30:34.330 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:30:34.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:30:34.331 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:30:34.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:30:34.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:30:34.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:30:34.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:30:34.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:30:34.432 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:30:34.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:30:34.867 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:30:34.867 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:30:34.867 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:30:34.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:30:34.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:34.868 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:30:35.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803838 2024-09-15T22:30:35.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803838 2024-09-15T22:30:35.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803838' 2024-09-15T22:30:35.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:35.110 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:30:35.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860419 2024-09-15T22:30:35.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860419 2024-09-15T22:30:35.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803838 1-412316860419' 2024-09-15T22:30:35.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:35.265 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:30:35.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449101 2024-09-15T22:30:35.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449101 2024-09-15T22:30:35.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803838 1-412316860419 2-335007449101' 2024-09-15T22:30:35.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:35.406 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:30:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084326 2024-09-15T22:30:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084326 2024-09-15T22:30:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803838 1-412316860419 2-335007449101 3-120259084326' 2024-09-15T22:30:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:35.562 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:35.562 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803838 2024-09-15T22:30:35.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:30:35.563 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803838 2024-09-15T22:30:35.563 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803838 2024-09-15T22:30:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803838' 2024-09-15T22:30:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803838 2024-09-15T22:30:35.565 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:30:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803838 -lt 25769803838 2024-09-15T22:30:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:35.988 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-412316860419 2024-09-15T22:30:35.988 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:35.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:30:35.991 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:35.991 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-412316860419 2024-09-15T22:30:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860419 2024-09-15T22:30:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 412316860419' 2024-09-15T22:30:35.993 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 412316860419 2024-09-15T22:30:35.993 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:30:36.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860419 -lt 412316860419 2024-09-15T22:30:36.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:36.439 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449101 2024-09-15T22:30:36.440 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:36.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:30:36.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449101 2024-09-15T22:30:36.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:36.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449101 2024-09-15T22:30:36.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449101' 2024-09-15T22:30:36.443 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 335007449101 2024-09-15T22:30:36.443 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:30:36.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449101 -lt 335007449101 2024-09-15T22:30:36.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:36.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:36.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084326 2024-09-15T22:30:36.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:30:36.901 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084326 2024-09-15T22:30:36.901 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:36.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084326 2024-09-15T22:30:36.902 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084326' 2024-09-15T22:30:36.902 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 120259084326 2024-09-15T22:30:36.903 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:30:37.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084326 -lt 120259084326 2024-09-15T22:30:37.366 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:30:37.367 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:30:37.367 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:30:37.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:30:37.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:30:37.954 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:30:37.954 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:30:37.955 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:30:37.955 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:30:37.955 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:30:37.955 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:30:38.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:30:38.381 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:30:38.381 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:30:38.381 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:30:38.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:30:38.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:30:38.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:30:38.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:30:38.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:30:39.692 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:30:39.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-1-256 2024-09-15T22:30:39.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:30:39.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:30:39.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-1-256 2024-09-15T22:30:39.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:30:39.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:30:39.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81867-1-256 td/test-erasure-eio/COPY 2024-09-15T22:30:39.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:30:39.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:30:39.870 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2024-09-15T22:30:39.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2024-09-15T22:30:39.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-81867-1-256 td/test-erasure-eio 2 256 add 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-81867-1-256 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-81867-1-256 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:30:39.873 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-81867-1-256 2024-09-15T22:30:39.874 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-81867-1-256 2024-09-15T22:30:39.874 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:30:40.285 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:30:40.285 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:30:40.285 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:30:40.285 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:30:40.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-15T22:30:40.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-15T22:30:40.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2024-09-15T22:30:40.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-15T22:30:41.080 INFO:tasks.workunit.client.0.smithi104.stderr:noout is set 2024-09-15T22:30:41.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-15T22:30:41.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-81867-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:41.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:30:41.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:30:41.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T22:30:41.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:30:41.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-81867-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:41.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:30:41.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:30:41.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T22:30:41.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:30:41.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T22:30:41.106 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:30:41.106 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:30:41.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:30:41.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:30:41.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:30:42.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:30:42.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-81867-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:42.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:30:42.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:30:42.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T22:30:42.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:30:42.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-15T22:30:42.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-81867-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:43.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-15T22:30:43.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:30:43.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:30:43.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T22:30:43.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:30:43.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:30:43.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:30:43.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:30:43.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:30:43.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:30:43.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:30:43.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:30:43.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:30:43.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:30:43.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:30:43.281 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:30:43.281 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:30:43.281 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:30:43.282 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:30:43.282 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:30:43.283 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:30:43.283 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:30:43.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:30:43.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:30:43.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:30:43.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:30:43.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:30:43.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:30:43.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:30:43.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:30:43.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:30:43.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:30:43.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:30:43.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T22:30:43.284 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.2 2024-09-15T22:30:43.284 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:30:43.285 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T22:30:43.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T22:30:43.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:30:43.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:30:43.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:30:43.318 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:43.316+0000 7fbacc20f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:43.320 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:43.320+0000 7fbacc20f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:43.322 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:43.320+0000 7fbacc20f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:43.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T22:30:43.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:30:43.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:30:43.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:30:43.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:30:43.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:43.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:30:43.729 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:30:43.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:43.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:30:44.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:44.386 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:44.384+0000 7fbacc20f7c0 -1 Falling back to public interface 2024-09-15T22:30:45.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:45.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:45.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:30:45.205 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:30:45.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:45.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:30:45.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:46.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:46.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:46.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:30:46.633 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:30:46.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:46.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:30:47.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:47.615 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:47.611+0000 7fbacc20f7c0 -1 osd.2 102 log_to_monitors true 2024-09-15T22:30:48.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:30:48.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:48.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:30:48.075 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:30:48.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:48.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:30:48.522 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 up in weight 1 up_from 107 up_thru 92 down_at 103 last_clean_interval [78,102) [v2:127.0.0.1:6818/3719790878,v1:127.0.0.1:6819/3719790878] [v2:127.0.0.1:6820/3719790878,v1:127.0.0.1:6821/3719790878] exists,up e7d01e95-a214-4f20-b1e3-81d56a7731dc 2024-09-15T22:30:48.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:30:48.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:30:48.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:30:48.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:30:48.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:30:48.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:30:48.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:30:48.523 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:30:48.524 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:30:48.524 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:30:48.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:30:48.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:30:48.525 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:30:48.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:30:48.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:30:48.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:30:48.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:30:48.634 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:30:48.634 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:30:49.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:30:49.452 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:30:49.452 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:30:49.452 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:30:49.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:30:49.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:49.452 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:30:49.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803843 2024-09-15T22:30:49.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803843 2024-09-15T22:30:49.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843' 2024-09-15T22:30:49.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:49.711 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:30:49.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860424 2024-09-15T22:30:49.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860424 2024-09-15T22:30:49.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-412316860424' 2024-09-15T22:30:49.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:49.864 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:30:50.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500675 2024-09-15T22:30:50.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500675 2024-09-15T22:30:50.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-412316860424 2-459561500675' 2024-09-15T22:30:50.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:30:50.374 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:30:50.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084331 2024-09-15T22:30:50.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084331 2024-09-15T22:30:50.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-412316860424 2-459561500675 3-120259084331' 2024-09-15T22:30:50.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:50.562 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803843 2024-09-15T22:30:50.563 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:50.563 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:30:50.564 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803843 2024-09-15T22:30:50.564 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:50.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803843 2024-09-15T22:30:50.565 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803843' 2024-09-15T22:30:50.565 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803843 2024-09-15T22:30:50.565 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:30:50.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803843 -lt 25769803843 2024-09-15T22:30:50.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:50.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-412316860424 2024-09-15T22:30:50.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:50.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:30:50.999 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-412316860424 2024-09-15T22:30:51.000 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:51.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860424 2024-09-15T22:30:51.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 412316860424' 2024-09-15T22:30:51.001 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 412316860424 2024-09-15T22:30:51.002 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:30:51.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860422 -lt 412316860424 2024-09-15T22:30:51.419 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:30:52.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:30:52.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:30:52.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860424 -lt 412316860424 2024-09-15T22:30:52.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:52.843 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-459561500675 2024-09-15T22:30:52.843 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:52.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:30:52.846 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-459561500675 2024-09-15T22:30:52.846 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:52.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500675 2024-09-15T22:30:52.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 459561500675' 2024-09-15T22:30:52.847 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 459561500675 2024-09-15T22:30:52.848 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:30:53.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500675 -lt 459561500675 2024-09-15T22:30:53.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:30:53.273 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084331 2024-09-15T22:30:53.273 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:30:53.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:30:53.276 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084331 2024-09-15T22:30:53.276 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:30:53.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084331 2024-09-15T22:30:53.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084331' 2024-09-15T22:30:53.278 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 120259084331 2024-09-15T22:30:53.278 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:30:53.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084331 -lt 120259084331 2024-09-15T22:30:53.710 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:30:53.711 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:30:53.711 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:30:54.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:30:54.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:30:54.322 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:30:54.322 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:30:54.322 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:30:54.322 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:30:54.322 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:30:54.322 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:30:54.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:30:54.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:30:54.744 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:30:54.744 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:30:55.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:30:55.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:30:55.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:30:55.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-15T22:30:55.317 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records in 2024-09-15T22:30:55.317 INFO:tasks.workunit.client.0.smithi104.stderr:1+0 records out 2024-09-15T22:30:55.317 INFO:tasks.workunit.client.0.smithi104.stderr:256 bytes copied, 4.3358e-05 s, 5.9 MB/s 2024-09-15T22:30:55.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-81867-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:55.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:30:55.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:30:55.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T22:30:55.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:30:55.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-81867-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:55.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:30:55.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:30:55.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T22:30:55.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:30:55.318 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T22:30:55.318 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:30:55.318 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:30:55.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:30:55.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:30:55.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:30:55.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:30:55.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-81867-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:55.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:30:55.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:30:55.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T22:30:55.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:30:55.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-15T22:30:55.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-81867-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T22:30:58.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-15T22:30:58.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:30:58.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:30:58.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T22:30:58.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:30:58.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:30:58.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:30:58.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:30:58.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:30:58.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:30:58.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:30:58.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:30:58.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:30:58.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:30:58.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:30:58.598 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:30:58.598 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:30:58.598 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:30:58.598 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:30:58.598 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:30:58.618 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:30:58.618 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:30:58.618 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.2 2024-09-15T22:30:58.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:30:58.620 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T22:30:58.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T22:30:58.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:30:58.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:30:58.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:30:58.629 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:58.627+0000 7f9b4cd817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:58.633 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:58.631+0000 7f9b4cd817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:58.634 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:58.631+0000 7f9b4cd817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:30:59.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T22:30:59.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:30:59.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:30:59.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:30:59.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:30:59.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:30:59.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:30:59.047 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:30:59.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:30:59.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:30:59.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:30:59.890 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:30:59.887+0000 7f9b4cd817c0 -1 Falling back to public interface 2024-09-15T22:31:00.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:00.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:00.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:31:00.529 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:31:00.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:00.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:31:00.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:01.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:01.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:01.958 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:31:01.958 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:31:01.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:01.959 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:31:02.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:02.869 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:02.867+0000 7f9b4cd817c0 -1 osd.2 109 log_to_monitors true 2024-09-15T22:31:03.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:03.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:03.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:31:03.482 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:31:03.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:03.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:31:04.319 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 up in weight 1 up_from 114 up_thru 92 down_at 110 last_clean_interval [107,109) [v2:127.0.0.1:6818/4292678701,v1:127.0.0.1:6819/4292678701] [v2:127.0.0.1:6820/4292678701,v1:127.0.0.1:6821/4292678701] exists,up e7d01e95-a214-4f20-b1e3-81d56a7731dc 2024-09-15T22:31:04.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:31:04.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:31:04.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:31:04.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:31:04.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:31:04.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:31:04.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:31:04.319 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:31:04.320 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:31:04.320 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:31:04.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:31:04.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:31:04.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:31:04.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:31:04.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:31:04.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:31:04.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:31:04.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:31:04.440 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:31:04.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:31:04.868 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:31:04.868 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:31:04.868 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:31:04.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:31:04.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:31:04.868 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:31:05.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803848 2024-09-15T22:31:05.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803848 2024-09-15T22:31:05.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848' 2024-09-15T22:31:05.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:31:05.161 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:31:05.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860429 2024-09-15T22:31:05.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860429 2024-09-15T22:31:05.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848 1-412316860429' 2024-09-15T22:31:05.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:31:05.317 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:31:05.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=489626271747 2024-09-15T22:31:05.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 489626271747 2024-09-15T22:31:05.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848 1-412316860429 2-489626271747' 2024-09-15T22:31:05.460 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:31:05.460 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:31:05.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084336 2024-09-15T22:31:05.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084336 2024-09-15T22:31:05.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803848 1-412316860429 2-489626271747 3-120259084336' 2024-09-15T22:31:05.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:31:05.611 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803848 2024-09-15T22:31:05.611 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:31:05.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:31:05.613 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803848 2024-09-15T22:31:05.613 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:31:05.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803848 2024-09-15T22:31:05.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803848' 2024-09-15T22:31:05.615 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803848 2024-09-15T22:31:05.615 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:31:06.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803848 -lt 25769803848 2024-09-15T22:31:06.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:31:06.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-412316860429 2024-09-15T22:31:06.055 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:31:06.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:31:06.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-412316860429 2024-09-15T22:31:06.058 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:31:06.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860429 2024-09-15T22:31:06.059 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 412316860429' 2024-09-15T22:31:06.059 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 412316860429 2024-09-15T22:31:06.059 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:31:06.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860429 -lt 412316860429 2024-09-15T22:31:06.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:31:06.534 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-489626271747 2024-09-15T22:31:06.534 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:31:06.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:31:06.536 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-489626271747 2024-09-15T22:31:06.536 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:31:06.537 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=489626271747 2024-09-15T22:31:06.537 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 489626271747' 2024-09-15T22:31:06.537 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 489626271747 2024-09-15T22:31:06.538 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:31:06.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271747 -lt 489626271747 2024-09-15T22:31:06.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:31:06.999 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:31:06.999 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084336 2024-09-15T22:31:07.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:31:07.001 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084336 2024-09-15T22:31:07.001 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:31:07.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084336 2024-09-15T22:31:07.002 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084336' 2024-09-15T22:31:07.002 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 120259084336 2024-09-15T22:31:07.003 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:31:07.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084336 -lt 120259084336 2024-09-15T22:31:07.491 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:31:07.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:31:07.492 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:31:08.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:31:08.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:31:08.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:31:08.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:31:08.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:31:08.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:31:08.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:31:08.121 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:31:08.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:31:08.509 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:31:08.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:31:08.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:31:09.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:31:09.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:31:09.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:31:09.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T22:31:09.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-15T22:31:09.843 INFO:tasks.workunit.client.0.smithi104.stderr:noout is unset 2024-09-15T22:31:09.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-15T22:31:09.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81867-1-256 2024-09-15T22:31:09.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:31:09.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:31:09.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81867-1-256 2024-09-15T22:31:09.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:31:09.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:31:09.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81867-1-256 td/test-erasure-eio/COPY 2024-09-15T22:31:10.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:31:10.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:31:10.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T22:31:10.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:346: TEST_rados_get_bad_size_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:31:10.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:31:10.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:31:10.645 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:31:10.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:31:11.350 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:31:11.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:31:11.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:31:11.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:31:11.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:31:11.368 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:31:11.368 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:31:11.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:31:11.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:31:11.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:31:11.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:31:11.682 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:31:11.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:31:11.684 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:31:11.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:31:11.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:31:11.686 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:31:11.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:31:11.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:31:11.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:31:11.689 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:31:11.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:31:11.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:31:11.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:31:11.727 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:31:11.727 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:31:11.727 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:31:11.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:31:11.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:31:11.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:31:11.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:31:11.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:31:11.733 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:31:11.734 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:31:11.734 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:31:11.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:31:11.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:31:11.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:31:11.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:31:11.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:31:11.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:31:11.739 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:31:11.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:31:11.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:31:11.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:31:11.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:31:11.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:31:11.743 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:31:11.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:31:11.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:31:11.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:31:11.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T22:31:11.745 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:31:11.746 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:31:11.746 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:31:11.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T22:31:11.748 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:31:11.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:31:11.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:31:11.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T22:31:11.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T22:31:11.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T22:31:11.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:31:11.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:31:11.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:31:11.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T22:31:11.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T22:31:12.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:31:12.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:31:12.244 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:31:12.245 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:31:12.245 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:31:12.245 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:31:12.245 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:31:12.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T22:31:12.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:31:12.455 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:31:12.455 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:31:12.455 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:31:12.455 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:31:12.456 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:31:12.456 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:31:12.456 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:31:12.456 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:31:12.456 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:31:12.457 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:31:12.457 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:31:12.457 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:31:12.457 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:31:12.457 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T22:31:12.573 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:31:12.573 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:31:12.573 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:31:12.573 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:31:12.573 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:31:12.574 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:31:12.574 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:31:12.574 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:31:12.575 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:31:12.575 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:31:12.575 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:31:12.575 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:31:12.575 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:31:12.575 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T22:31:12.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T22:31:12.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T22:31:12.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:31:12.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:31:12.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:31:12.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T22:31:12.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:31:13.577 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:31:13.577 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:31:13.577 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:31:13.577 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:31:13.577 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:31:13.577 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:31:13.577 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:31:13.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:31:13.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:31:13.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T22:31:13.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T22:31:13.900 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T22:31:13.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:31:14.918 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T22:31:14.918 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:31:14.918 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:31:14.919 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:31:14.919 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:31:14.919 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:31:14.919 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:31:14.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T22:31:14.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T22:31:15.020 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:31:15.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T22:31:15.036 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:31:12.287+0000 7f5fb4527e00 0 load: jerasure load: lrc 2024-09-15T22:31:15.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_0 td/test-erasure-eio 2024-09-15T22:31:15.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:275: TEST_rados_get_subread_eio_shard_0: local dir=td/test-erasure-eio 2024-09-15T22:31:15.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:276: TEST_rados_get_subread_eio_shard_0: setup_osds 4 2024-09-15T22:31:15.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T22:31:15.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T22:31:15.038 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T22:31:15.040 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T22:31:15.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:31:15.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T22:31:15.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:31:15.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:31:15.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:31:15.044 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:31:15.044 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:31:15.044 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:31:15.044 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:31:15.044 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:31:15.044 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:31:15.044 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:31:15.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:31:15.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:31:15.045 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:31:15.046 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3a1b3671-ba21-486f-9dbf-c5d74143fb62 2024-09-15T22:31:15.047 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 3a1b3671-ba21-486f-9dbf-c5d74143fb62 2024-09-15T22:31:15.047 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3a1b3671-ba21-486f-9dbf-c5d74143fb62' 2024-09-15T22:31:15.047 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:31:15.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAzYOdmF6fEAxAAQMzQ88fRlESDTDtznDlaFw== 2024-09-15T22:31:15.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAzYOdmF6fEAxAAQMzQ88fRlESDTDtznDlaFw=="}' 2024-09-15T22:31:15.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3a1b3671-ba21-486f-9dbf-c5d74143fb62 -i td/test-erasure-eio/0/new.json 2024-09-15T22:31:15.434 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:31:15.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T22:31:15.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzYOdmF6fEAxAAQMzQ88fRlESDTDtznDlaFw== --osd-uuid 3a1b3671-ba21-486f-9dbf-c5d74143fb62 2024-09-15T22:31:15.480 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:15.479+0000 7feaa301c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:15.503 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:15.499+0000 7feaa301c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:15.504 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:15.503+0000 7feaa301c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:15.504 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:15.503+0000 7feaa301c7c0 -1 bdev(0x556d7aef5c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:31:15.504 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:15.503+0000 7feaa301c7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T22:31:21.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T22:31:21.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:31:21.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:31:21.672 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T22:31:21.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:31:22.245 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T22:31:22.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:31:22.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:31:22.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:31:22.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:31:22.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:31:22.278 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:22.275+0000 7f40ae6997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:22.289 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:22.287+0000 7f40ae6997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:22.291 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:22.287+0000 7f40ae6997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:22.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:31:22.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:31:22.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:31:22.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:31:22.708 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:31:22.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:22.709 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:31:22.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:31:22.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:22.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:23.163 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:23.338 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:23.335+0000 7f40ae6997c0 -1 Falling back to public interface 2024-09-15T22:31:24.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:24.165 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:24.165 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:31:24.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:31:24.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:24.166 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:24.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:25.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:25.637 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:25.638 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:31:25.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:31:25.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:25.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:25.958 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:25.955+0000 7f40ae6997c0 -1 osd.0 0 log_to_monitors true 2024-09-15T22:31:26.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:27.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:27.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:27.070 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:31:27.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:31:27.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:27.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:27.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:28.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:28.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:28.523 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:31:28.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:31:28.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:28.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:28.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:29.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:29.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:29.955 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:31:29.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:31:29.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:29.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:30.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:31.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:31.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:31.413 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:31:31.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:31:31.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:31.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:31.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:32.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:32.946 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:32.947 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:31:32.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:31:32.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:32.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:33.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:34.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:34.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:34.379 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:31:34.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:31:34.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:34.380 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:34.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:35.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:35.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:35.919 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:31:35.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:31:35.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:35.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:36.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:37.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:37.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:37.401 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:31:37.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:31:37.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:37.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:37.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:38.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:38.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:38.908 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:31:38.908 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:31:38.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:38.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:39.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:40.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:40.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:40.395 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:31:40.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:31:40.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:40.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:41.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:42.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:42.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:42.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:31:42.024 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:31:42.025 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:42.025 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:42.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:43.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:43.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:43.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:31:43.771 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:31:43.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:43.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:44.935 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:45.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:45.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:45.937 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:31:45.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:31:45.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:45.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:46.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:47.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:47.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:47.955 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:31:47.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:31:47.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:47.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:48.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:49.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:49.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:49.722 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:31:49.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:31:49.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:49.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:31:50.248 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/853244277,v1:127.0.0.1:6803/853244277] [v2:127.0.0.1:6804/853244277,v1:127.0.0.1:6805/853244277] exists,up 3a1b3671-ba21-486f-9dbf-c5d74143fb62 2024-09-15T22:31:50.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:31:50.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:31:50.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:31:50.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:31:50.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T22:31:50.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:31:50.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:31:50.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:31:50.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:31:50.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:31:50.250 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:31:50.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:31:50.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:31:50.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:31:50.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:31:50.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:31:50.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:31:50.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:31:50.251 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:31:50.251 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:31:50.251 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:31:50.251 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:31:50.253 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:31:50.253 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:31:50.253 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:31:50.253 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:31:50.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:31:50.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:31:50.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:31:50.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:31:50.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:31:50.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:31:50.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:31:50.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:31:50.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:31:50.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:31:50.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:31:50.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:31:50.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:31:50.255 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:31:50.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a715f332-bc17-45d4-a744-76d86070f085 2024-09-15T22:31:50.255 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 a715f332-bc17-45d4-a744-76d86070f085 2024-09-15T22:31:50.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a715f332-bc17-45d4-a744-76d86070f085' 2024-09-15T22:31:50.256 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:31:50.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBWYOdm2aR2EBAAOsf8rnxL0YnSssLUe6rwOQ== 2024-09-15T22:31:50.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBWYOdm2aR2EBAAOsf8rnxL0YnSssLUe6rwOQ=="}' 2024-09-15T22:31:50.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a715f332-bc17-45d4-a744-76d86070f085 -i td/test-erasure-eio/1/new.json 2024-09-15T22:31:50.939 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:31:50.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T22:31:50.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWYOdm2aR2EBAAOsf8rnxL0YnSssLUe6rwOQ== --osd-uuid a715f332-bc17-45d4-a744-76d86070f085 2024-09-15T22:31:50.992 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:50.991+0000 7f8dbad817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:51.009 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:51.007+0000 7f8dbad817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:51.010 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:51.007+0000 7f8dbad817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:51.010 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:51.007+0000 7f8dbad817c0 -1 bdev(0x5569a4177c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:31:51.011 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:51.007+0000 7f8dbad817c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T22:31:56.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T22:31:56.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:31:56.875 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T22:31:56.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:31:56.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:31:57.435 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T22:31:57.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:31:57.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:31:57.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:31:57.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:31:57.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:31:57.468 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:57.467+0000 7f8d5d17c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:57.486 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:57.483+0000 7f8d5d17c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:57.487 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:57.483+0000 7f8d5d17c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:31:57.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:31:57.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:31:57.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:31:57.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:31:57.856 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:31:57.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:31:57.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:57.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:31:57.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:57.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:31:58.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:31:58.771 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:31:58.767+0000 7f8d5d17c7c0 -1 Falling back to public interface 2024-09-15T22:31:59.293 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:31:59.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:31:59.294 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:31:59.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:31:59.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:31:59.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:31:59.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:00.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:00.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:00.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:32:00.761 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:32:00.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:00.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:01.190 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:01.676 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:32:01.675+0000 7f8d5d17c7c0 -1 osd.1 0 log_to_monitors true 2024-09-15T22:32:02.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:02.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:02.192 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:32:02.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:32:02.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:02.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:02.851 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:03.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:03.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:03.853 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:32:03.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:32:03.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:03.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:04.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:05.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:05.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:05.277 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:32:05.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:32:05.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:05.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:05.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:06.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:06.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:06.724 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:32:06.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:32:06.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:06.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:07.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:08.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:08.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:08.182 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:32:08.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:32:08.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:08.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:08.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:09.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:09.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:09.858 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:32:09.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:32:09.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:09.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:10.289 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:11.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:11.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:11.291 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:32:11.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:32:11.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:11.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:11.848 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:12.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:12.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:12.850 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:32:12.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:32:12.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:12.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:13.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:14.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:14.290 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:14.290 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:32:14.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:32:14.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:14.291 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:15.323 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:16.329 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:16.329 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:32:16.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:32:16.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:16.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:16.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:17.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:32:17.845 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:32:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:17.845 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:18.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:19.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:19.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:19.642 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:32:19.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:32:19.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:19.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:20.539 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:21.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:21.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:21.542 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:32:21.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:32:21.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:21.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:22.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:23.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:23.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:23.280 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:32:23.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:32:23.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:23.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:24.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:25.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:25.377 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:25.378 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:32:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:32:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:25.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:26.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:27.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:27.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:27.008 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:32:27.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:32:27.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:27.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:32:27.580 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/533750909,v1:127.0.0.1:6811/533750909] [v2:127.0.0.1:6812/533750909,v1:127.0.0.1:6813/533750909] exists,up a715f332-bc17-45d4-a744-76d86070f085 2024-09-15T22:32:27.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:32:27.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:32:27.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:32:27.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:32:27.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T22:32:27.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:32:27.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:32:27.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:32:27.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:32:27.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:32:27.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:32:27.585 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:32:27.585 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:32:27.585 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:32:27.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:32:27.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:32:27.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:32:27.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:32:27.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:32:27.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:32:27.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:32:27.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:32:27.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:32:27.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:32:27.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:32:27.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:32:27.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:32:27.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:32:27.586 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:32:27.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ae1aa33b-61b2-4788-8718-97fa9d477ea4 2024-09-15T22:32:27.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 ae1aa33b-61b2-4788-8718-97fa9d477ea4' 2024-09-15T22:32:27.588 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 ae1aa33b-61b2-4788-8718-97fa9d477ea4 2024-09-15T22:32:27.588 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:32:27.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB7YOdm8MNTJBAA3BO9vFe5DT0A874clQuT4Q== 2024-09-15T22:32:27.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB7YOdm8MNTJBAA3BO9vFe5DT0A874clQuT4Q=="}' 2024-09-15T22:32:27.610 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ae1aa33b-61b2-4788-8718-97fa9d477ea4 -i td/test-erasure-eio/2/new.json 2024-09-15T22:32:28.111 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:32:28.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T22:32:28.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB7YOdm8MNTJBAA3BO9vFe5DT0A874clQuT4Q== --osd-uuid ae1aa33b-61b2-4788-8718-97fa9d477ea4 2024-09-15T22:32:28.164 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:32:28.162+0000 7fe7a29a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:32:28.166 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:32:28.162+0000 7fe7a29a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:32:28.167 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:32:28.162+0000 7fe7a29a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:32:28.168 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:32:28.166+0000 7fe7a29a77c0 -1 bdev(0x55d929599c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:32:28.168 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:32:28.166+0000 7fe7a29a77c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T22:32:34.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T22:32:34.874 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:32:34.875 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:32:34.876 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T22:32:34.876 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:32:35.436 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:32:35.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:32:35.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:32:35.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:32:35.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:32:35.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:32:35.470 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:32:35.466+0000 7f0978a137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:32:35.485 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:32:35.482+0000 7f0978a137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:32:35.487 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:32:35.482+0000 7f0978a137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:32:35.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:32:35.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:32:35.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:32:35.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:32:35.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:32:35.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:35.917 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:32:35.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:32:35.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:35.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:36.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:36.534 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:32:36.530+0000 7f0978a137c0 -1 Falling back to public interface 2024-09-15T22:32:37.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:37.415 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:37.416 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:32:37.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:32:37.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:37.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:37.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:38.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:38.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:38.793 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:32:38.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:32:38.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:38.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:39.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:39.367 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:32:39.350+0000 7f0978a137c0 -1 osd.2 0 log_to_monitors true 2024-09-15T22:32:40.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:40.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:40.213 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:32:40.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:32:40.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:40.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:41.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:42.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:42.007 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:42.008 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:32:42.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:32:42.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:42.008 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:42.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:43.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:43.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:43.606 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:32:43.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:32:43.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:43.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:44.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:45.068 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:45.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:45.069 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:32:45.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:32:45.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:45.070 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:45.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:46.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:46.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:46.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:32:46.693 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:32:46.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:46.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:47.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:48.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:48.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:48.272 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:32:48.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:32:48.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:48.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:48.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:49.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:49.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:49.764 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:32:49.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:32:49.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:49.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:50.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:51.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:51.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:51.269 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:32:51.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:32:51.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:51.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:51.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:52.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:52.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:52.749 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:32:52.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:32:52.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:52.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:53.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:54.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:54.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:54.663 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:32:54.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:32:54.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:54.664 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:55.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:56.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:56.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:56.525 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:32:56.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:32:56.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:56.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:57.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:32:58.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:32:58.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:32:58.605 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:32:58.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:32:58.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:32:58.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:32:59.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:00.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:00.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:00.469 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:33:00.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:33:00.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:00.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:33:01.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:02.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:02.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:02.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:33:02.220 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:33:02.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:02.220 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:33:03.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:04.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:04.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:04.104 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:33:04.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:33:04.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:04.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:33:04.589 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3258864735,v1:127.0.0.1:6819/3258864735] [v2:127.0.0.1:6820/3258864735,v1:127.0.0.1:6821/3258864735] exists,up ae1aa33b-61b2-4788-8718-97fa9d477ea4 2024-09-15T22:33:04.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:33:04.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:33:04.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:33:04.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:33:04.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T22:33:04.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:33:04.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:33:04.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T22:33:04.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:33:04.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:33:04.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:33:04.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:33:04.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:33:04.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:33:04.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:33:04.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:33:04.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:33:04.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:33:04.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:33:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:33:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:33:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:33:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:33:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:33:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:33:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:33:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:33:04.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:33:04.597 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:33:04.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=50500659-f225-4da2-ba20-a2d4320eaa8b 2024-09-15T22:33:04.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 50500659-f225-4da2-ba20-a2d4320eaa8b' 2024-09-15T22:33:04.598 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 50500659-f225-4da2-ba20-a2d4320eaa8b 2024-09-15T22:33:04.598 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:33:04.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCgYOdmN+SJJBAAb9nLCSXcvjAEKrXDpzZfcQ== 2024-09-15T22:33:04.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCgYOdmN+SJJBAAb9nLCSXcvjAEKrXDpzZfcQ=="}' 2024-09-15T22:33:04.614 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 50500659-f225-4da2-ba20-a2d4320eaa8b -i td/test-erasure-eio/3/new.json 2024-09-15T22:33:05.757 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:33:05.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T22:33:05.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgYOdmN+SJJBAAb9nLCSXcvjAEKrXDpzZfcQ== --osd-uuid 50500659-f225-4da2-ba20-a2d4320eaa8b 2024-09-15T22:33:05.804 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:05.802+0000 7fe3e8d2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:33:05.877 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:05.874+0000 7fe3e8d2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:33:05.879 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:05.874+0000 7fe3e8d2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:33:05.879 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:05.874+0000 7fe3e8d2b7c0 -1 bdev(0x559323207c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T22:33:05.879 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:05.878+0000 7fe3e8d2b7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T22:33:12.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T22:33:12.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:33:12.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T22:33:12.870 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T22:33:12.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:33:13.458 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T22:33:13.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T22:33:13.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:33:13.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:33:13.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:33:13.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:33:13.490 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:13.486+0000 7fd1fc11e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:33:13.498 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:13.494+0000 7fd1fc11e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:33:13.499 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:13.494+0000 7fd1fc11e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:33:13.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T22:33:13.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:33:13.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:33:13.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:33:13.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:33:13.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:13.913 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:33:13.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:33:13.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:13.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:14.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:15.226 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:15.222+0000 7fd1fc11e7c0 -1 Falling back to public interface 2024-09-15T22:33:15.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:15.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:15.445 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:33:15.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:33:15.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:15.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:15.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:16.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:16.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:16.934 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:33:16.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:33:16.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:16.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:17.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:17.985 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:17.982+0000 7fd1fc11e7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T22:33:18.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:18.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:18.378 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:33:18.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:33:18.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:18.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:19.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:20.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:20.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:20.029 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:33:20.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:33:20.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:20.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:20.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:21.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:21.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:21.604 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:33:21.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:33:21.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:21.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:22.204 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:23.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:23.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:23.206 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:33:23.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:33:23.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:23.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:23.777 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:24.778 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:24.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:24.779 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:33:24.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:33:24.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:24.780 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:25.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:26.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:26.270 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:26.270 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:33:26.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:33:26.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:26.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:26.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:27.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:27.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:27.798 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:33:27.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:33:27.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:27.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:28.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:29.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:29.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:29.274 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:33:29.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:33:29.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:29.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:29.812 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:30.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:30.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:30.815 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:33:30.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:33:30.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:30.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:31.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:32.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:32.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:32.511 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:33:32.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:33:32.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:32.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:33.825 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:34.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:34.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:34.827 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:33:34.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:33:34.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:34.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:36.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:36.558 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:36.558 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:33:36.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:33:36.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:36.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:37.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:38.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:38.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:38.064 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:33:38.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:33:38.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:38.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:38.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:39.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:39.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:39.822 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:33:39.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:33:39.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:39.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:40.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:41.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:41.555 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:41.555 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:33:41.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:33:41.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:41.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:42.033 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:33:43.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:33:43.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:33:43.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:33:43.035 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:33:43.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:33:43.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:33:43.691 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3799508039,v1:127.0.0.1:6827/3799508039] [v2:127.0.0.1:6828/3799508039,v1:127.0.0.1:6829/3799508039] exists,up 50500659-f225-4da2-ba20-a2d4320eaa8b 2024-09-15T22:33:43.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:33:43.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:33:43.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:33:43.693 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:33:43.693 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:33:43.693 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:33:43.693 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:33:43.693 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:43.693 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:43.694 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:33:43.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:33:43.694 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:33:43.809 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:33:43.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:33:43.827 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:31:23.347+0000 7f40ae6997c0 0 load: jerasure load: lrc 2024-09-15T22:33:43.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:278: TEST_rados_get_subread_eio_shard_0: local poolname=pool-jerasure 2024-09-15T22:33:43.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:279: TEST_rados_get_subread_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T22:33:43.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:33:43.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:33:43.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T22:33:43.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:33:43.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T22:33:43.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:33:43.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T22:33:44.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:33:44.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:33:45.928 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:33:45.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:33:46.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:33:46.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:33:46.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:33:46.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:33:46.950 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:33:46.951 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:33:46.951 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:33:46.952 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:33:46.952 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:33:46.952 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:33:47.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:33:47.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:33:47.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:33:47.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:33:47.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:33:47.083 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:33:47.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:33:47.580 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:33:47.580 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:33:47.580 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:33:47.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:33:47.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:33:47.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:33:47.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803812 2024-09-15T22:33:47.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803812 2024-09-15T22:33:47.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812' 2024-09-15T22:33:47.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:33:47.755 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:33:47.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542168 2024-09-15T22:33:47.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542168 2024-09-15T22:33:47.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-60129542168' 2024-09-15T22:33:47.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:33:47.907 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:33:48.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345932 2024-09-15T22:33:48.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345932 2024-09-15T22:33:48.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-60129542168 2-85899345932' 2024-09-15T22:33:48.063 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:33:48.063 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:33:48.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=133143986183 2024-09-15T22:33:48.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 133143986183 2024-09-15T22:33:48.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-60129542168 2-85899345932 3-133143986183' 2024-09-15T22:33:48.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:33:48.218 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803812 2024-09-15T22:33:48.218 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:33:48.219 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:33:48.220 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803812 2024-09-15T22:33:48.220 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:33:48.221 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803812 2024-09-15T22:33:48.221 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803812 2024-09-15T22:33:48.222 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-15T22:33:48.222 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:33:48.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-15T22:33:48.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:33:48.640 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-60129542168 2024-09-15T22:33:48.640 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:33:48.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:33:48.642 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-60129542168 2024-09-15T22:33:48.642 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:33:48.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542168 2024-09-15T22:33:48.644 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 60129542168 2024-09-15T22:33:48.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 60129542168' 2024-09-15T22:33:48.644 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:33:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542168 -lt 60129542168 2024-09-15T22:33:49.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:33:49.115 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345932 2024-09-15T22:33:49.115 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:33:49.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:33:49.118 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345932 2024-09-15T22:33:49.118 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:33:49.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345932 2024-09-15T22:33:49.119 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345932' 2024-09-15T22:33:49.119 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 85899345932 2024-09-15T22:33:49.120 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:33:49.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345932 -lt 85899345932 2024-09-15T22:33:49.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:33:49.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-133143986183 2024-09-15T22:33:49.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:33:49.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:33:49.587 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:33:49.587 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-133143986183 2024-09-15T22:33:49.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=133143986183 2024-09-15T22:33:49.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 133143986183' 2024-09-15T22:33:49.589 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 133143986183 2024-09-15T22:33:49.589 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:33:50.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 133143986183 -lt 133143986183 2024-09-15T22:33:50.197 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:33:50.197 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:33:50.197 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:33:50.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:33:50.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:33:50.804 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:33:50.804 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:33:50.804 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:33:50.804 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:33:50.805 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:33:50.805 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:33:51.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:33:51.231 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:33:51.232 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:33:51.232 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:33:51.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:33:51.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:33:51.806 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:281: TEST_rados_get_subread_eio_shard_0: local shard_id=0 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:282: TEST_rados_get_subread_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-81867-0 2024-09-15T22:33:51.807 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-81867-0 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-81867-0 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:33:51.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-81867-0 td/test-erasure-eio/ORIGINAL 2024-09-15T22:33:51.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81867-0 td/test-erasure-eio 0 2024-09-15T22:33:51.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:33:51.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:33:51.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:33:51.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:33:51.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:33:51.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:33:51.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-81867-0 2024-09-15T22:33:51.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:33:51.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:33:51.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:33:51.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T22:33:51.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:33:51.974 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-81867-0 2024-09-15T22:33:51.974 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:33:51.974 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-81867-0 2024-09-15T22:33:51.975 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-81867-0 2024-09-15T22:33:51.975 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:33:52.482 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:33:52.482 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:33:52.482 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:33:52.482 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:33:52.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T22:33:52.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:33:52.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-15T22:33:52.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:33:52.483 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T22:33:52.483 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:33:52.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T22:33:52.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:33:52.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-15T22:33:52.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:33:52.484 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:33:52.485 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:33:52.485 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-15T22:33:52.485 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:33:52.485 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:33:52.486 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:33:52.486 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:52.486 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:52.486 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:33:52.486 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:33:52.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:33:52.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:33:52.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:33:52.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-15T22:33:52.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:33:52.600 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:33:52.600 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:33:52.600 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:52.601 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:52.601 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:33:52.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:33:52.601 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-81867-0 0 2024-09-15T22:33:52.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81867-0 2024-09-15T22:33:52.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:33:52.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:33:52.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81867-0 2024-09-15T22:33:52.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:33:52.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:33:52.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-81867-0 td/test-erasure-eio/COPY 2024-09-15T22:33:52.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:33:52.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:33:52.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-15T22:33:52.761 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 0 + 1 2024-09-15T22:33:52.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=1 2024-09-15T22:33:52.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81867-0 td/test-erasure-eio 1 2024-09-15T22:33:52.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:33:52.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:33:52.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:33:52.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:33:52.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:33:52.764 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:33:52.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-81867-0 2024-09-15T22:33:52.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:33:52.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:33:52.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:33:52.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-15T22:33:52.765 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:33:52.765 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-81867-0 2024-09-15T22:33:52.765 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:33:52.765 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-81867-0 2024-09-15T22:33:52.765 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-81867-0 2024-09-15T22:33:52.765 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:33:53.192 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:33:53.192 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:33:53.192 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:33:53.192 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:33:53.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T22:33:53.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:33:53.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-15T22:33:53.193 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:33:53.194 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-15T22:33:53.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:33:53.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T22:33:53.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:33:53.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T22:33:53.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:33:53.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:33:53.195 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:33:53.195 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T22:33:53.195 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T22:33:53.195 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T22:33:53.196 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:33:53.196 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:53.197 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:53.197 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.1.asok 2024-09-15T22:33:53.197 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:33:53.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:33:53.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:33:53.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:33:53.319 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-15T22:33:53.319 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T22:33:53.319 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T22:33:53.320 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:33:53.320 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:53.320 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:53.320 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.1.asok 2024-09-15T22:33:53.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:33:53.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-81867-0 1 2024-09-15T22:33:53.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-15T22:33:53.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81867-0 fail 2024-09-15T22:33:53.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:33:53.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:33:53.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81867-0 2024-09-15T22:33:53.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T22:33:53.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T22:33:53.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-81867-0 td/test-erasure-eio/COPY 2024-09-15T22:33:53.462 INFO:tasks.workunit.client.0.smithi104.stderr:error getting pool-jerasure/obj-eio-81867-0: (5) Input/output error 2024-09-15T22:33:53.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T22:33:53.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-15T22:33:53.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:283: TEST_rados_get_subread_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:33:53.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:33:53.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:33:53.995 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:33:54.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:33:54.799 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:33:54.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:33:54.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:33:54.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:33:54.819 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:33:54.820 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:33:54.820 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:33:54.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:33:54.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:33:54.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:33:55.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:33:55.135 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:33:55.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:33:55.138 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:33:55.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:33:55.140 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:33:55.140 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:33:55.141 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:33:55.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:33:55.142 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:33:55.143 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:33:55.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:33:55.145 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:33:55.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:33:55.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:33:55.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:55.172 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:55.172 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:33:55.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:33:55.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:33:55.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T22:33:55.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T22:33:55.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T22:33:55.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T22:33:55.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:33:55.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:33:55.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:33:55.175 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:33:55.175 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:33:55.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:33:55.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:33:55.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:33:55.177 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:33:55.177 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:33:55.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:33:55.179 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:33:55.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:33:55.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:33:55.180 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:33:55.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:33:55.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:33:55.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:33:55.182 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:33:55.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:33:55.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:33:55.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:33:55.186 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:33:55.186 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:55.186 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:55.186 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:33:55.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:33:55.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:33:55.187 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T22:33:55.189 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:33:55.189 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:55.189 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:55.189 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T22:33:55.191 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:33:55.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:33:55.191 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:33:55.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T22:33:55.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T22:33:55.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T22:33:55.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:33:55.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:33:55.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:33:55.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T22:33:55.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T22:33:55.661 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:33:55.662 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:33:55.662 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:33:55.662 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:33:55.662 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:55.663 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:55.663 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:33:55.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T22:33:55.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:33:55.874 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:33:55.875 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:33:55.875 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:33:55.875 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:33:55.875 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:33:55.876 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:33:55.876 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:33:55.876 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:33:55.876 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:33:55.876 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:55.876 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:55.877 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:33:55.877 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:33:55.877 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T22:33:55.995 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:33:55.995 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:33:55.995 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:33:55.995 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:33:55.996 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:33:55.996 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:33:55.996 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:33:55.996 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:33:55.997 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:33:55.997 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:55.997 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:55.997 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:33:55.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:33:55.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T22:33:56.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T22:33:56.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T22:33:56.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:33:56.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:33:56.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:33:56.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T22:33:56.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:33:57.048 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:33:57.048 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:33:57.049 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:33:57.049 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:33:57.049 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:57.050 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:57.050 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:33:57.050 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:33:57.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:33:57.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T22:33:57.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T22:33:57.434 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T22:33:57.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:33:58.453 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T22:33:58.454 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:33:58.454 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:33:58.454 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:33:58.455 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:58.455 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:58.455 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:33:58.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T22:33:58.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T22:33:58.563 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:33:58.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T22:33:58.580 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:33:55.701+0000 7f4913b15e00 0 load: jerasure load: lrc 2024-09-15T22:33:58.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_1 td/test-erasure-eio 2024-09-15T22:33:58.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:287: TEST_rados_get_subread_eio_shard_1: local dir=td/test-erasure-eio 2024-09-15T22:33:58.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:288: TEST_rados_get_subread_eio_shard_1: setup_osds 4 2024-09-15T22:33:58.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T22:33:58.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T22:33:58.582 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T22:33:58.583 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T22:33:58.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:33:58.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T22:33:58.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:33:58.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:33:58.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:33:58.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:33:58.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T22:33:58.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:33:58.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:33:58.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:33:58.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:33:58.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T22:33:58.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T22:33:58.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:33:58.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:33:58.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:33:58.587 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:33:58.587 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:33:58.587 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:33:58.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:33:58.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:33:58.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:33:58.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:33:58.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:33:58.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T22:33:58.590 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:33:58.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a2890d88-58b4-4bff-bb81-749567b8b24a 2024-09-15T22:33:58.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a2890d88-58b4-4bff-bb81-749567b8b24a' 2024-09-15T22:33:58.591 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 a2890d88-58b4-4bff-bb81-749567b8b24a 2024-09-15T22:33:58.591 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:33:58.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDWYOdmkHuTJBAAXIYIo1/I0RphbLZnxtOveg== 2024-09-15T22:33:58.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDWYOdmkHuTJBAAXIYIo1/I0RphbLZnxtOveg=="}' 2024-09-15T22:33:58.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a2890d88-58b4-4bff-bb81-749567b8b24a -i td/test-erasure-eio/0/new.json 2024-09-15T22:33:58.901 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:33:58.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T22:33:58.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWYOdmkHuTJBAAXIYIo1/I0RphbLZnxtOveg== --osd-uuid a2890d88-58b4-4bff-bb81-749567b8b24a 2024-09-15T22:33:58.955 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:58.949+0000 7fe39d1db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:33:58.970 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:58.965+0000 7fe39d1db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:33:58.972 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:58.969+0000 7fe39d1db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:33:58.972 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:58.969+0000 7fe39d1db7c0 -1 bdev(0x55cfa86afc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:33:58.973 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:33:58.969+0000 7fe39d1db7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T22:34:05.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T22:34:05.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:34:05.039 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T22:34:05.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:34:05.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:34:05.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:34:05.606 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T22:34:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:34:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:34:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:34:05.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:34:05.637 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:05.633+0000 7f892b84c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:34:05.658 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:05.653+0000 7f892b84c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:34:05.659 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:05.653+0000 7f892b84c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:34:06.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:34:06.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:34:06.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:34:06.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:34:06.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:34:06.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:06.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:34:06.079 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:34:06.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:06.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:06.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:06.930 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:06.925+0000 7f892b84c7c0 -1 Falling back to public interface 2024-09-15T22:34:07.553 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:07.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:07.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:34:07.554 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:34:07.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:07.554 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:07.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:08.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:08.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:08.972 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:34:08.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:34:08.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:08.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:09.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:09.737 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:09.729+0000 7f892b84c7c0 -1 osd.0 0 log_to_monitors true 2024-09-15T22:34:10.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:10.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:10.408 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:34:10.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:34:10.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:10.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:10.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:11.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:11.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:11.871 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:34:11.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:34:11.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:11.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:12.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:13.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:13.321 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:13.321 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:34:13.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:34:13.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:13.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:13.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:14.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:14.864 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:14.864 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:34:14.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:34:14.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:14.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:15.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:16.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:16.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:16.313 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:34:16.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:34:16.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:16.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:16.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:17.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:17.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:17.830 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:34:17.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:34:17.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:17.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:18.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:19.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:19.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:19.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:34:19.280 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:34:19.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:19.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:19.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:20.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:20.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:20.865 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:34:20.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:34:20.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:20.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:21.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:22.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:22.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:22.314 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:34:22.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:34:22.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:22.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:23.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:24.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:24.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:24.297 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:34:24.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:34:24.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:24.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:25.171 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:26.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:26.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:26.173 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:34:26.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:34:26.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:26.174 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:27.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:28.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:28.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:28.255 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:34:28.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:34:28.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:28.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:28.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:29.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:29.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:29.837 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:34:29.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:34:29.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:29.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:30.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:31.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:31.872 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:31.872 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:34:31.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:34:31.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:31.873 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:32.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:33.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:33.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:33.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:34:33.821 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:34:33.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:33.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:34:34.310 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1270059232,v1:127.0.0.1:6803/1270059232] [v2:127.0.0.1:6804/1270059232,v1:127.0.0.1:6805/1270059232] exists,up a2890d88-58b4-4bff-bb81-749567b8b24a 2024-09-15T22:34:34.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:34:34.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:34:34.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:34:34.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:34:34.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T22:34:34.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:34:34.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:34:34.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:34:34.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:34:34.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:34:34.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:34:34.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:34:34.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:34:34.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:34:34.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:34:34.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:34:34.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:34:34.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:34:34.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:34:34.315 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:34:34.315 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:34:34.315 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:34:34.315 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:34:34.315 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:34:34.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:34:34.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:34:34.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:34:34.317 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:34:34.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fea81b59-31a8-48b3-bc0f-e0b3bb268c46 2024-09-15T22:34:34.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 fea81b59-31a8-48b3-bc0f-e0b3bb268c46' 2024-09-15T22:34:34.318 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 fea81b59-31a8-48b3-bc0f-e0b3bb268c46 2024-09-15T22:34:34.318 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:34:34.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD6YOdmG1YoFBAAhqNJexlLFlONIt683CwQWA== 2024-09-15T22:34:34.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD6YOdmG1YoFBAAhqNJexlLFlONIt683CwQWA=="}' 2024-09-15T22:34:34.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fea81b59-31a8-48b3-bc0f-e0b3bb268c46 -i td/test-erasure-eio/1/new.json 2024-09-15T22:34:34.828 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:34:34.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T22:34:34.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6YOdmG1YoFBAAhqNJexlLFlONIt683CwQWA== --osd-uuid fea81b59-31a8-48b3-bc0f-e0b3bb268c46 2024-09-15T22:34:34.871 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:34.865+0000 7fb4106577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:34:34.899 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:34.893+0000 7fb4106577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:34:34.900 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:34.897+0000 7fb4106577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:34:34.900 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:34.897+0000 7fb4106577c0 -1 bdev(0x555d88db8800 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:34:34.900 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:34.897+0000 7fb4106577c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T22:34:41.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T22:34:41.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:34:41.132 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T22:34:41.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:34:41.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:34:41.720 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T22:34:41.720 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:34:41.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:34:41.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:34:41.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:34:41.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:34:41.751 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:41.745+0000 7ff24632d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:34:41.759 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:41.753+0000 7ff24632d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:34:41.760 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:41.757+0000 7ff24632d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:34:42.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:34:42.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:34:42.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:34:42.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:34:42.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:34:42.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:42.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:34:42.155 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:34:42.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:42.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:42.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:42.822 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:42.817+0000 7ff24632d7c0 -1 Falling back to public interface 2024-09-15T22:34:43.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:43.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:43.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:34:43.660 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:34:43.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:43.661 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:44.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:45.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:45.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:45.089 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:34:45.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:34:45.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:45.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:45.486 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:34:45.481+0000 7ff24632d7c0 -1 osd.1 0 log_to_monitors true 2024-09-15T22:34:45.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:46.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:46.526 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:46.526 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:34:46.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:34:46.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:46.527 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:47.090 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:48.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:48.091 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:48.092 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:34:48.092 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:34:48.092 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:48.092 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:48.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:49.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:49.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:49.550 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:34:49.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:34:49.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:49.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:49.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:50.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:50.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:50.970 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:34:50.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:34:50.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:50.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:51.531 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:52.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:52.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:52.533 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:34:52.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:34:52.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:52.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:52.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:53.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:53.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:53.965 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:34:53.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:34:53.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:53.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:54.529 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:55.531 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:55.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:55.532 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:34:55.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:34:55.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:55.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:55.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:56.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:56.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:56.978 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:34:56.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:34:56.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:56.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:57.530 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:34:58.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:34:58.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:34:58.532 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:34:58.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:34:58.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:34:58.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:34:59.299 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:00.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:00.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:00.301 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:35:00.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:35:00.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:00.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:35:00.779 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:01.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:01.781 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:01.781 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:35:01.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:35:01.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:01.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:35:03.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:04.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:04.296 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:04.296 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:35:04.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:35:04.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:04.297 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:35:04.992 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:05.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:05.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:05.994 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:35:05.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:35:05.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:05.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:35:06.588 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:07.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:07.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:07.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:35:07.590 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:35:07.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:07.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:35:08.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:09.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:09.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:09.197 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:35:09.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:35:09.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:09.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:35:09.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:10.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:10.762 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:10.763 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:35:10.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:35:10.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:10.763 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:35:11.257 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/887364809,v1:127.0.0.1:6811/887364809] [v2:127.0.0.1:6812/887364809,v1:127.0.0.1:6813/887364809] exists,up fea81b59-31a8-48b3-bc0f-e0b3bb268c46 2024-09-15T22:35:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:35:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:35:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:35:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:35:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T22:35:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:35:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:35:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:35:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:35:11.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:35:11.260 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:35:11.262 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:35:11.262 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:35:11.262 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:35:11.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:35:11.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:35:11.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:35:11.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:35:11.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:35:11.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:35:11.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:35:11.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:35:11.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:35:11.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:35:11.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:35:11.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:35:11.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:35:11.264 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:35:11.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8449d7b4-e455-489f-ae97-f4e986408434 2024-09-15T22:35:11.264 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 8449d7b4-e455-489f-ae97-f4e986408434 2024-09-15T22:35:11.265 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8449d7b4-e455-489f-ae97-f4e986408434' 2024-09-15T22:35:11.265 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:35:11.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAfYedmgOr1EBAAgkqjnOuwnXFGrQZ+6Swl7A== 2024-09-15T22:35:11.285 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAfYedmgOr1EBAAgkqjnOuwnXFGrQZ+6Swl7A=="}' 2024-09-15T22:35:11.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8449d7b4-e455-489f-ae97-f4e986408434 -i td/test-erasure-eio/2/new.json 2024-09-15T22:35:11.799 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:35:11.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T22:35:11.816 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfYedmgOr1EBAAgkqjnOuwnXFGrQZ+6Swl7A== --osd-uuid 8449d7b4-e455-489f-ae97-f4e986408434 2024-09-15T22:35:11.845 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:11.840+0000 7fd9a78f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:11.873 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:11.868+0000 7fd9a78f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:11.874 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:11.868+0000 7fd9a78f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:11.874 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:11.868+0000 7fd9a78f37c0 -1 bdev(0x5643d30f7c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:35:11.875 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:11.868+0000 7fd9a78f37c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T22:35:17.995 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T22:35:17.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:35:17.997 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T22:35:17.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:35:17.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:35:18.575 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:35:18.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:35:18.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:35:18.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:35:18.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:35:18.576 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:35:18.607 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:18.600+0000 7f6ce2f6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:18.619 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:18.612+0000 7f6ce2f6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:18.620 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:18.616+0000 7f6ce2f6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:19.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:35:19.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:35:19.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:35:19.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:35:19.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:35:19.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:19.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:35:19.049 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:35:19.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:19.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:19.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:20.098 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:20.092+0000 7f6ce2f6d7c0 -1 Falling back to public interface 2024-09-15T22:35:20.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:20.480 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:20.480 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:35:20.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:35:20.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:20.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:20.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:21.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:21.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:21.945 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:35:21.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:35:21.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:21.945 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:22.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:23.016 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:23.012+0000 7f6ce2f6d7c0 -1 osd.2 0 log_to_monitors true 2024-09-15T22:35:23.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:23.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:23.403 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:35:23.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:35:23.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:23.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:23.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:24.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:24.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:24.857 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:35:24.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:35:24.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:24.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:25.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:26.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:26.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:26.308 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:35:26.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:35:26.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:26.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:27.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:28.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:28.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:28.022 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:35:28.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:35:28.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:28.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:28.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:29.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:29.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:29.493 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:35:29.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:35:29.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:29.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:30.003 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:31.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:31.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:31.005 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:35:31.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:35:31.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:31.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:31.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:32.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:32.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:32.464 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:35:32.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:35:32.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:32.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:33.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:34.050 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:34.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:34.051 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:35:34.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:35:34.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:34.051 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:34.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:35.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:35.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:35.585 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:35:35.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:35:35.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:35.586 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:36.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:37.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:37.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:37.022 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:35:37.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:35:37.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:37.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:37.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:38.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:38.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:38.837 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:35:38.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:35:38.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:38.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:39.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:40.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:40.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:40.940 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:35:40.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:35:40.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:40.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:42.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:43.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:43.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:43.138 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:35:43.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:35:43.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:43.139 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:43.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:44.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:44.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:44.971 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:35:44.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:35:44.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:44.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:45.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:46.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:46.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:46.860 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:35:46.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:35:46.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:46.861 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:47.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:48.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:35:48.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:48.870 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:35:48.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:35:48.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:48.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:35:49.360 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/756241322,v1:127.0.0.1:6819/756241322] [v2:127.0.0.1:6820/756241322,v1:127.0.0.1:6821/756241322] exists,up 8449d7b4-e455-489f-ae97-f4e986408434 2024-09-15T22:35:49.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:35:49.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:35:49.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:35:49.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:35:49.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T22:35:49.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:35:49.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:35:49.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T22:35:49.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:35:49.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:35:49.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:35:49.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:35:49.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:35:49.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:35:49.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:35:49.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:35:49.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:35:49.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:35:49.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:35:49.364 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:35:49.364 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:35:49.364 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:35:49.365 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:35:49.365 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:35:49.365 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:35:49.365 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:35:49.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:35:49.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:35:49.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:35:49.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:35:49.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:35:49.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:35:49.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:35:49.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:35:49.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:35:49.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:35:49.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:35:49.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:35:49.366 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:35:49.366 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:35:49.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d4ee50de-12fc-41a8-ae70-af0c5e085343 2024-09-15T22:35:49.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 d4ee50de-12fc-41a8-ae70-af0c5e085343' 2024-09-15T22:35:49.367 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 d4ee50de-12fc-41a8-ae70-af0c5e085343 2024-09-15T22:35:49.367 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:35:49.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBFYedmVqz2FhAAefrTibsKocnRemyQ8s/enw== 2024-09-15T22:35:49.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBFYedmVqz2FhAAefrTibsKocnRemyQ8s/enw=="}' 2024-09-15T22:35:49.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d4ee50de-12fc-41a8-ae70-af0c5e085343 -i td/test-erasure-eio/3/new.json 2024-09-15T22:35:50.642 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:35:50.662 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T22:35:50.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFYedmVqz2FhAAefrTibsKocnRemyQ8s/enw== --osd-uuid d4ee50de-12fc-41a8-ae70-af0c5e085343 2024-09-15T22:35:50.698 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:50.692+0000 7f8d146d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:50.731 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:50.728+0000 7f8d146d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:50.733 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:50.728+0000 7f8d146d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:50.733 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:50.728+0000 7f8d146d47c0 -1 bdev(0x5566a46a4800 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T22:35:50.734 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:50.728+0000 7f8d146d47c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T22:35:57.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T22:35:57.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:35:57.654 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T22:35:57.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T22:35:57.654 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:35:58.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T22:35:58.286 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T22:35:58.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:35:58.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:35:58.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:35:58.288 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:35:58.319 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:58.312+0000 7f2be4b5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:58.323 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:58.316+0000 7f2be4b5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:58.325 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:58.320+0000 7f2be4b5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:35:58.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T22:35:58.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:35:58.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:35:58.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:35:58.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:35:58.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:35:58.693 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:35:58.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:35:58.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:35:58.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:35:59.109 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:35:59.570 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:35:59.564+0000 7f2be4b5c7c0 -1 Falling back to public interface 2024-09-15T22:36:00.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:00.110 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:00.111 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:36:00.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:36:00.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:00.111 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:00.581 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:01.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:01.583 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:01.583 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:36:01.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:36:01.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:01.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:02.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:02.475 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:36:02.468+0000 7f2be4b5c7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T22:36:03.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:03.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:03.013 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:36:03.013 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:36:03.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:03.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:03.850 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:04.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:04.853 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:04.853 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:36:04.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:36:04.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:04.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:05.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:06.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:06.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:06.341 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:36:06.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:36:06.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:06.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:06.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:07.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:07.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:07.871 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:36:07.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:36:07.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:07.871 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:08.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:09.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:09.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:09.471 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:36:09.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:36:09.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:09.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:09.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:10.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:10.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:10.896 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:36:10.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:36:10.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:10.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:11.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:12.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:12.451 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:12.452 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:36:12.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:36:12.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:12.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:12.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:13.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:13.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:13.976 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:36:13.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:36:13.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:13.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:14.496 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:15.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:15.497 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:15.498 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:36:15.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:36:15.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:15.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:15.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:16.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:16.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:16.928 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:36:16.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:36:16.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:16.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:17.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:18.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:18.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:18.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:36:18.810 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:36:18.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:18.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:19.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:20.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:20.701 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:20.701 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:36:20.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:36:20.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:20.702 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:21.800 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:22.801 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:22.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:22.802 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:36:22.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:36:22.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:22.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:23.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:24.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:24.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:24.534 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:36:24.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:36:24.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:24.535 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:25.406 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:26.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:26.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:26.408 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:36:26.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:36:26.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:26.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:27.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:28.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:28.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:28.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:36:28.348 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:36:28.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:28.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:36:29.084 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/533350769,v1:127.0.0.1:6827/533350769] [v2:127.0.0.1:6828/533350769,v1:127.0.0.1:6829/533350769] exists,up d4ee50de-12fc-41a8-ae70-af0c5e085343 2024-09-15T22:36:29.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:36:29.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:36:29.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:36:29.085 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:36:29.085 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:36:29.085 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:36:29.086 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:36:29.086 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:29.086 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:29.086 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:36:29.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:36:29.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:36:29.193 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:36:29.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:36:29.212 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:34:06.937+0000 7f892b84c7c0 0 load: jerasure load: lrc 2024-09-15T22:36:29.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:290: TEST_rados_get_subread_eio_shard_1: local poolname=pool-jerasure 2024-09-15T22:36:29.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:291: TEST_rados_get_subread_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T22:36:29.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:36:29.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:36:29.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T22:36:29.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:36:29.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T22:36:29.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:36:29.214 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T22:36:30.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:36:30.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:36:31.143 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:36:31.159 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:36:32.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:36:32.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:36:32.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:36:32.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:36:32.162 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:36:32.163 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:36:32.163 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:36:32.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:36:32.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:36:32.164 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:36:32.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:36:32.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:36:32.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:36:32.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:36:32.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:36:32.293 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:36:32.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:36:32.739 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:36:32.739 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:36:32.739 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:36:32.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:36:32.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:36:32.739 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:36:32.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803803 2024-09-15T22:36:32.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803803 2024-09-15T22:36:32.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803803' 2024-09-15T22:36:32.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:36:32.921 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:36:33.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574874 2024-09-15T22:36:33.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574874 2024-09-15T22:36:33.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803803 1-55834574874' 2024-09-15T22:36:33.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:36:33.155 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:36:33.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345936 2024-09-15T22:36:33.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345936 2024-09-15T22:36:33.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803803 1-55834574874 2-85899345936' 2024-09-15T22:36:33.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:36:33.307 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:36:33.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018889 2024-09-15T22:36:33.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018889 2024-09-15T22:36:33.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803803 1-55834574874 2-85899345936 3-128849018889' 2024-09-15T22:36:33.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:36:33.462 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803803 2024-09-15T22:36:33.462 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:36:33.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:36:33.463 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803803 2024-09-15T22:36:33.464 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:36:33.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803803 2024-09-15T22:36:33.465 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803803 2024-09-15T22:36:33.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803803' 2024-09-15T22:36:33.466 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:36:33.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803801 -lt 25769803803 2024-09-15T22:36:33.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:36:34.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:36:34.905 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:36:35.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803804 -lt 25769803803 2024-09-15T22:36:35.333 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:36:35.334 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:36:35.334 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574874 2024-09-15T22:36:35.335 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:36:35.336 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574874 2024-09-15T22:36:35.336 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:36:35.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574874 2024-09-15T22:36:35.338 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574874 2024-09-15T22:36:35.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574874' 2024-09-15T22:36:35.338 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:36:35.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574874 -lt 55834574874 2024-09-15T22:36:35.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:36:35.819 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345936 2024-09-15T22:36:35.819 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:36:35.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:36:35.821 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345936 2024-09-15T22:36:35.821 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:36:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345936 2024-09-15T22:36:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345936' 2024-09-15T22:36:35.823 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 85899345936 2024-09-15T22:36:35.823 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:36:36.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345937 -lt 85899345936 2024-09-15T22:36:36.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:36:36.338 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:36:36.338 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018889 2024-09-15T22:36:36.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:36:36.341 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018889 2024-09-15T22:36:36.341 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:36:36.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018889 2024-09-15T22:36:36.342 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 128849018889 2024-09-15T22:36:36.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018889' 2024-09-15T22:36:36.343 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:36:36.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018889 -lt 128849018889 2024-09-15T22:36:36.831 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:36:36.831 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:36:36.832 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:36:37.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:36:37.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:36:37.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:36:37.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:36:37.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:36:37.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:36:37.422 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:36:37.423 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:36:37.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:36:37.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:36:37.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:36:37.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:36:38.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:36:38.420 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:293: TEST_rados_get_subread_eio_shard_1: local shard_id=1 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:294: TEST_rados_get_subread_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-15T22:36:38.421 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-81867-1 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-81867-1 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-81867-1 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:36:38.422 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-81867-1 td/test-erasure-eio/ORIGINAL 2024-09-15T22:36:38.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81867-1 td/test-erasure-eio 1 2024-09-15T22:36:38.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:36:38.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:36:38.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:36:38.590 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:36:38.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:36:38.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:36:38.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-81867-1 2024-09-15T22:36:38.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:36:38.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:36:38.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:36:38.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-15T22:36:38.591 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:36:38.591 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-81867-1 2024-09-15T22:36:38.591 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:36:38.591 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-81867-1 2024-09-15T22:36:38.592 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:36:38.592 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-81867-1 2024-09-15T22:36:39.010 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:36:39.010 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:36:39.010 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:36:39.010 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:36:39.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T22:36:39.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:36:39.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-15T22:36:39.010 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:36:39.011 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-15T22:36:39.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:36:39.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T22:36:39.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:36:39.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T22:36:39.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:36:39.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:36:39.013 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:36:39.013 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T22:36:39.013 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T22:36:39.013 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T22:36:39.013 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:36:39.014 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:39.014 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:39.014 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.1.asok 2024-09-15T22:36:39.014 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:36:39.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:36:39.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:36:39.130 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:36:39.131 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-15T22:36:39.131 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T22:36:39.131 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T22:36:39.131 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:36:39.131 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:39.131 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:39.132 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.1.asok 2024-09-15T22:36:39.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:36:39.132 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-81867-1 1 2024-09-15T22:36:39.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81867-1 2024-09-15T22:36:39.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:36:39.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:36:39.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81867-1 2024-09-15T22:36:39.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:36:39.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:36:39.246 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-81867-1 td/test-erasure-eio/COPY 2024-09-15T22:36:39.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:36:39.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:36:39.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-15T22:36:39.284 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-15T22:36:39.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=2 2024-09-15T22:36:39.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81867-1 td/test-erasure-eio 2 2024-09-15T22:36:39.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:36:39.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:36:39.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:36:39.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:36:39.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:36:39.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:36:39.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-81867-1 2024-09-15T22:36:39.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:36:39.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:36:39.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:36:39.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=2 2024-09-15T22:36:39.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:36:39.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-81867-1 2024-09-15T22:36:39.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:36:39.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-81867-1 2024-09-15T22:36:39.287 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-81867-1 2024-09-15T22:36:39.287 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:36:39.736 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:36:39.737 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:36:39.737 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:36:39.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:36:39.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T22:36:39.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:36:39.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=2 2024-09-15T22:36:39.737 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:36:39.737 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/2/type 2024-09-15T22:36:39.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:36:39.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-15T22:36:39.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:36:39.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=2 2024-09-15T22:36:39.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:36:39.739 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:36:39.740 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:36:39.740 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.2 2024-09-15T22:36:39.740 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-15T22:36:39.740 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-15T22:36:39.741 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:36:39.741 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:39.741 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:39.741 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.2.asok 2024-09-15T22:36:39.741 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:36:39.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:36:39.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:36:39.859 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.2 2024-09-15T22:36:39.859 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-15T22:36:39.859 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-15T22:36:39.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:36:39.859 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:36:39.859 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:39.859 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:39.859 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.2.asok 2024-09-15T22:36:39.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:36:39.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.2.asok injectdataerr pool-jerasure obj-eio-81867-1 2 2024-09-15T22:36:39.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-15T22:36:39.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:188: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81867-1 2024-09-15T22:36:39.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:36:39.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:36:39.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81867-1 2024-09-15T22:36:39.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:36:39.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:36:39.984 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-81867-1 td/test-erasure-eio/COPY 2024-09-15T22:36:40.016 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:36:40.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:36:40.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-15T22:36:40.018 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:295: TEST_rados_get_subread_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:36:40.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:36:40.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:36:40.628 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:36:40.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:36:41.310 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:36:41.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:36:41.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:36:41.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:36:41.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:36:41.327 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:36:41.327 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:36:41.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:36:41.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:36:41.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:36:41.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:36:41.438 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:36:41.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:36:41.439 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:36:41.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:36:41.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:36:41.441 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:36:41.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:36:41.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:36:41.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:36:41.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:36:41.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:36:41.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:36:41.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:36:41.462 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:36:41.462 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:41.462 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:41.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:36:41.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:36:41.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:36:41.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T22:36:41.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T22:36:41.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T22:36:41.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T22:36:41.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:36:41.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:36:41.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:36:41.465 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:36:41.465 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:36:41.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:36:41.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:36:41.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:36:41.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:36:41.466 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:36:41.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:36:41.467 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:36:41.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:36:41.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:36:41.468 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:36:41.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:36:41.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:36:41.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:36:41.470 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:36:41.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:36:41.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:36:41.471 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:36:41.472 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:36:41.472 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:41.472 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:41.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:36:41.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:36:41.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:36:41.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T22:36:41.474 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:36:41.475 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:41.475 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:41.475 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T22:36:41.476 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:36:41.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:36:41.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:36:41.476 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T22:36:41.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T22:36:41.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T22:36:41.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:36:41.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:36:41.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:36:41.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T22:36:41.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T22:36:41.941 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:36:41.941 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:36:41.942 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:36:41.942 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:36:41.942 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:41.942 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:41.943 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:36:41.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T22:36:42.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:36:42.126 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:36:42.126 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:36:42.126 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:36:42.126 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:36:42.127 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:36:42.127 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:36:42.127 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:36:42.127 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:36:42.127 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:36:42.127 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:42.128 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:42.128 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:36:42.128 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:36:42.128 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T22:36:42.238 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:36:42.239 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:36:42.239 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:36:42.239 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:36:42.239 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:36:42.239 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:36:42.239 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:36:42.240 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:36:42.240 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:36:42.240 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:42.240 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:42.240 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:36:42.241 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:36:42.241 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T22:36:42.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T22:36:42.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T22:36:42.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:36:42.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:36:42.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:36:42.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T22:36:42.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:36:43.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:36:43.250 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:36:43.250 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:36:43.250 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:36:43.250 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:43.250 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:43.251 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:36:43.252 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:36:43.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:36:43.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T22:36:43.287 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T22:36:43.564 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T22:36:43.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:36:44.583 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T22:36:44.583 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:36:44.583 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:36:44.584 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:36:44.584 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:44.584 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:44.584 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:36:44.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T22:36:44.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T22:36:44.686 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:36:44.699 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T22:36:44.702 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:36:41.983+0000 7f29d565ee00 0 load: jerasure load: lrc 2024-09-15T22:36:44.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_missing td/test-erasure-eio 2024-09-15T22:36:44.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_subread_missing: local dir=td/test-erasure-eio 2024-09-15T22:36:44.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_subread_missing: setup_osds 4 2024-09-15T22:36:44.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T22:36:44.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T22:36:44.704 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T22:36:44.706 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T22:36:44.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T22:36:44.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:36:44.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:36:44.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:36:44.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:36:44.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:36:44.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:36:44.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:36:44.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:36:44.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:36:44.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:36:44.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:36:44.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T22:36:44.713 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:36:44.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=be6ba97a-6264-449a-9963-6ef053b7b40a 2024-09-15T22:36:44.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 be6ba97a-6264-449a-9963-6ef053b7b40a' 2024-09-15T22:36:44.713 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 be6ba97a-6264-449a-9963-6ef053b7b40a 2024-09-15T22:36:44.714 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:36:44.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB8YedmI75vKxAA9dRKZZtEAsr8bNu+5F6Daw== 2024-09-15T22:36:44.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB8YedmI75vKxAA9dRKZZtEAsr8bNu+5F6Daw=="}' 2024-09-15T22:36:44.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new be6ba97a-6264-449a-9963-6ef053b7b40a -i td/test-erasure-eio/0/new.json 2024-09-15T22:36:45.040 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:36:45.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T22:36:45.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8YedmI75vKxAA9dRKZZtEAsr8bNu+5F6Daw== --osd-uuid be6ba97a-6264-449a-9963-6ef053b7b40a 2024-09-15T22:36:45.090 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:36:45.087+0000 7fbc206237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:36:45.117 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:36:45.115+0000 7fbc206237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:36:45.119 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:36:45.115+0000 7fbc206237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:36:45.119 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:36:45.115+0000 7fbc206237c0 -1 bdev(0x55cfa6f03c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:36:45.119 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:36:45.119+0000 7fbc206237c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T22:36:50.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T22:36:50.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:36:50.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:36:50.771 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T22:36:50.771 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:36:51.361 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T22:36:51.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:36:51.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:36:51.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:36:51.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:36:51.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:36:51.393 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:36:51.391+0000 7f86156077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:36:51.404 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:36:51.403+0000 7f86156077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:36:51.405 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:36:51.403+0000 7f86156077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:36:51.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:36:51.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:36:51.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:36:51.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:36:51.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:36:51.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:51.859 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:36:51.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:36:51.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:51.859 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:36:52.349 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:52.914 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:36:52.911+0000 7f86156077c0 -1 Falling back to public interface 2024-09-15T22:36:53.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:53.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:53.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:36:53.352 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:36:53.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:53.352 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:36:53.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:54.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:54.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:54.787 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:36:54.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:36:54.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:54.787 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:36:55.215 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:55.788 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:36:55.787+0000 7f86156077c0 -1 osd.0 0 log_to_monitors true 2024-09-15T22:36:56.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:56.217 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:56.218 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:36:56.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:36:56.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:56.218 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:36:56.852 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:57.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:57.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:57.854 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:36:57.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:36:57.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:57.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:36:58.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:36:59.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:36:59.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:36:59.283 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:36:59.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:36:59.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:36:59.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:36:59.750 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:00.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:00.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:00.752 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:37:00.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:37:00.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:00.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:01.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:02.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:02.182 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:02.182 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:37:02.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:37:02.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:02.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:02.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:03.681 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:03.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:03.759 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:37:03.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:37:03.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:03.759 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:04.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:05.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:05.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:05.123 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:37:05.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:37:05.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:05.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:05.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:06.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:06.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:06.596 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:37:06.596 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:37:06.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:06.597 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:07.052 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:08.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:08.053 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:37:08.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:08.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:37:08.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:08.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:09.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:10.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:10.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:10.023 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:37:10.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:37:10.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:10.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:10.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:11.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:11.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:11.953 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:37:11.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:37:11.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:11.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:13.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:14.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:14.389 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:14.390 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:37:14.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:37:14.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:14.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:14.840 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:15.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:15.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:15.842 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:37:15.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:37:15.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:15.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:16.338 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:17.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:17.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:17.340 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:37:17.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:37:17.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:17.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:18.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:19.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:19.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:19.077 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:37:19.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:37:19.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:19.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:37:19.643 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1339993899,v1:127.0.0.1:6803/1339993899] [v2:127.0.0.1:6804/1339993899,v1:127.0.0.1:6805/1339993899] exists,up be6ba97a-6264-449a-9963-6ef053b7b40a 2024-09-15T22:37:19.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:37:19.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:37:19.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:37:19.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:37:19.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:37:19.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:37:19.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:37:19.647 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:37:19.647 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:37:19.647 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:37:19.647 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:37:19.647 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:37:19.647 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:37:19.649 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:37:19.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:37:19.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:37:19.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:37:19.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:37:19.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:37:19.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:37:19.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:37:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:37:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:37:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:37:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:37:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:37:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:37:19.650 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:37:19.650 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9e0f8581-4b76-4b0d-93d0-b170ba1a5bbc 2024-09-15T22:37:19.650 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 9e0f8581-4b76-4b0d-93d0-b170ba1a5bbc 2024-09-15T22:37:19.651 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 9e0f8581-4b76-4b0d-93d0-b170ba1a5bbc' 2024-09-15T22:37:19.651 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:37:19.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCfYedmG/T4JxAAX20jj3p/G+zEGEQ4XNvHpw== 2024-09-15T22:37:19.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCfYedmG/T4JxAAX20jj3p/G+zEGEQ4XNvHpw=="}' 2024-09-15T22:37:19.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9e0f8581-4b76-4b0d-93d0-b170ba1a5bbc -i td/test-erasure-eio/1/new.json 2024-09-15T22:37:20.489 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:37:20.507 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T22:37:20.508 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfYedmG/T4JxAAX20jj3p/G+zEGEQ4XNvHpw== --osd-uuid 9e0f8581-4b76-4b0d-93d0-b170ba1a5bbc 2024-09-15T22:37:20.544 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:20.539+0000 7fecfe3877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:37:20.554 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:20.551+0000 7fecfe3877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:37:20.555 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:20.551+0000 7fecfe3877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:37:20.555 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:20.555+0000 7fecfe3877c0 -1 bdev(0x56289ee2dc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:37:20.555 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:20.555+0000 7fecfe3877c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T22:37:26.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T22:37:26.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:37:26.511 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T22:37:26.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:37:26.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:37:27.083 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T22:37:27.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:37:27.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:37:27.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:37:27.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:37:27.084 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:37:27.115 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:27.111+0000 7fb1538767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:37:27.141 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:27.139+0000 7fb1538767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:37:27.141 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:27.139+0000 7fb1538767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:37:27.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:37:27.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:37:27.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:37:27.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:37:27.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:37:27.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:27.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:37:27.513 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:37:27.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:27.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:27.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:28.598 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:28.595+0000 7fb1538767c0 -1 Falling back to public interface 2024-09-15T22:37:28.935 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:28.935 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:28.935 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:37:28.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:37:28.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:28.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:29.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:30.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:30.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:30.376 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:37:30.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:37:30.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:30.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:30.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:31.394 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:31.391+0000 7fb1538767c0 -1 osd.1 0 log_to_monitors true 2024-09-15T22:37:31.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:31.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:31.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:37:31.796 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:37:31.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:31.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:32.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:33.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:33.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:33.279 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:37:33.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:37:33.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:33.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:33.749 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:34.751 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:34.751 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:34.751 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:37:34.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:37:34.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:34.752 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:35.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:36.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:36.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:36.279 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:37:36.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:37:36.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:36.280 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:36.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:37.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:37.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:37.729 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:37:37.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:37:37.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:37.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:38.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:39.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:39.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:39.279 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:37:39.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:37:39.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:39.279 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:39.733 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:40.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:40.735 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:40.736 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:37:40.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:37:40.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:40.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:41.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:42.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:42.326 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:42.326 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:37:42.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:37:42.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:42.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:42.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:43.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:43.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:43.796 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:37:43.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:37:43.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:43.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:44.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:45.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:45.489 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:45.489 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:37:45.490 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:37:45.490 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:45.490 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:46.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:47.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:47.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:47.325 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:37:47.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:37:47.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:47.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:48.738 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:49.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:49.740 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:49.741 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:37:49.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:37:49.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:49.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:50.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:51.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:51.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:51.545 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:37:51.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:37:51.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:51.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:52.559 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:53.561 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:53.561 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:37:53.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:53.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:37:53.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:53.562 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:54.300 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:37:55.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:37:55.302 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:37:55.302 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:37:55.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:37:55.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:37:55.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:37:55.834 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/535676997,v1:127.0.0.1:6811/535676997] [v2:127.0.0.1:6812/535676997,v1:127.0.0.1:6813/535676997] exists,up 9e0f8581-4b76-4b0d-93d0-b170ba1a5bbc 2024-09-15T22:37:55.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:37:55.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:37:55.836 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:37:55.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:37:55.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:37:55.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:37:55.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:37:55.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:37:55.841 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:37:55.841 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:37:55.842 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:37:55.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:37:55.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:37:55.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:37:55.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:37:55.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:37:55.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:37:55.843 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:37:55.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=94ccd8ff-6423-4e72-98c5-8f82d4412245 2024-09-15T22:37:55.843 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 94ccd8ff-6423-4e72-98c5-8f82d4412245 2024-09-15T22:37:55.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 94ccd8ff-6423-4e72-98c5-8f82d4412245' 2024-09-15T22:37:55.844 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:37:55.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDDYedmzuYzMxAAHKxJ6h6h2pBm5QO5clZFIg== 2024-09-15T22:37:55.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDDYedmzuYzMxAAHKxJ6h6h2pBm5QO5clZFIg=="}' 2024-09-15T22:37:55.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 94ccd8ff-6423-4e72-98c5-8f82d4412245 -i td/test-erasure-eio/2/new.json 2024-09-15T22:37:56.545 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:37:56.566 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T22:37:56.567 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDYedmzuYzMxAAHKxJ6h6h2pBm5QO5clZFIg== --osd-uuid 94ccd8ff-6423-4e72-98c5-8f82d4412245 2024-09-15T22:37:56.602 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:56.598+0000 7f9590edf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:37:56.604 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:56.602+0000 7f9590edf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:37:56.605 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:56.602+0000 7f9590edf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:37:56.606 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:56.602+0000 7f9590edf7c0 -1 bdev(0x55cff918dc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:37:56.606 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:37:56.602+0000 7f9590edf7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T22:38:03.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T22:38:03.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:38:03.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:38:03.256 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T22:38:03.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:38:03.822 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:38:03.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:38:03.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:38:03.823 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:38:03.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:38:03.824 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:38:03.854 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:03.850+0000 7f02428617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:38:03.880 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:03.878+0000 7f02428617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:38:03.881 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:03.878+0000 7f02428617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:38:04.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:38:04.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:38:04.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:38:04.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:38:04.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:38:04.275 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:04.275 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:38:04.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:38:04.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:04.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:04.715 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:04.882 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:04.878+0000 7f02428617c0 -1 Falling back to public interface 2024-09-15T22:38:05.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:05.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:05.717 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:38:05.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:38:05.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:05.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:06.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:07.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:07.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:07.197 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:38:07.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:38:07.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:07.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:07.528 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:07.526+0000 7f02428617c0 -1 osd.2 0 log_to_monitors true 2024-09-15T22:38:07.629 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:08.630 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:08.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:08.631 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:38:08.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:38:08.631 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:08.632 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:09.235 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:10.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:10.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:10.237 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:38:10.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:38:10.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:10.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:10.795 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:11.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:11.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:11.797 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:38:11.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:38:11.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:11.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:12.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:13.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:13.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:13.254 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:38:13.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:38:13.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:13.255 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:13.791 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:14.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:14.793 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:14.793 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:38:14.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:38:14.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:14.794 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:15.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:16.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:16.233 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:16.233 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:38:16.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:38:16.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:16.234 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:16.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:17.914 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:17.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:17.915 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:38:17.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:38:17.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:17.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:18.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:19.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:19.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:19.453 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:38:19.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:38:19.453 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:19.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:19.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:20.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:20.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:20.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:38:20.912 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:38:20.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:20.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:21.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:22.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:22.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:22.341 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:38:22.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:38:22.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:22.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:23.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:24.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:24.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:24.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:38:24.196 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:38:24.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:24.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:24.918 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:25.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:25.919 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:25.920 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:38:25.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:38:25.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:25.920 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:26.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:27.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:27.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:27.374 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:38:27.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:38:27.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:27.375 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:28.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:29.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:29.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:29.205 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:38:29.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:38:29.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:29.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:30.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:31.208 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:31.208 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:31.209 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:38:31.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:38:31.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:31.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:31.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:32.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:32.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:32.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:38:32.717 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:38:32.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:32.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:38:33.297 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/664351535,v1:127.0.0.1:6819/664351535] [v2:127.0.0.1:6820/664351535,v1:127.0.0.1:6821/664351535] exists,up 94ccd8ff-6423-4e72-98c5-8f82d4412245 2024-09-15T22:38:33.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:38:33.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:38:33.305 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:38:33.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:38:33.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T22:38:33.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:38:33.306 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:38:33.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T22:38:33.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:38:33.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:38:33.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:38:33.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:38:33.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:38:33.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:38:33.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:38:33.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:38:33.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:38:33.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:38:33.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:38:33.309 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:38:33.309 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:38:33.309 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:38:33.309 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:38:33.309 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:38:33.310 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:38:33.310 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:38:33.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:38:33.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:38:33.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:38:33.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:38:33.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:38:33.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:38:33.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:38:33.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:38:33.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:38:33.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:38:33.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:38:33.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:38:33.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:38:33.312 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:38:33.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4d033fea-3ac4-4865-bfa1-1c7aa5cf1680 2024-09-15T22:38:33.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 4d033fea-3ac4-4865-bfa1-1c7aa5cf1680' 2024-09-15T22:38:33.313 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 4d033fea-3ac4-4865-bfa1-1c7aa5cf1680 2024-09-15T22:38:33.314 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:38:33.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDpYedmBckKExAAH4o+kNCf4dnXJrhtVYi4DQ== 2024-09-15T22:38:33.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDpYedmBckKExAAH4o+kNCf4dnXJrhtVYi4DQ=="}' 2024-09-15T22:38:33.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4d033fea-3ac4-4865-bfa1-1c7aa5cf1680 -i td/test-erasure-eio/3/new.json 2024-09-15T22:38:33.883 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:38:33.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T22:38:33.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpYedmBckKExAAH4o+kNCf4dnXJrhtVYi4DQ== --osd-uuid 4d033fea-3ac4-4865-bfa1-1c7aa5cf1680 2024-09-15T22:38:33.936 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:33.934+0000 7f6e63e827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:38:33.939 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:33.934+0000 7f6e63e827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:38:33.940 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:33.938+0000 7f6e63e827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:38:33.940 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:33.938+0000 7f6e63e827c0 -1 bdev(0x55a9222bdc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T22:38:33.940 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:33.938+0000 7f6e63e827c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T22:38:40.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T22:38:40.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:38:40.688 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T22:38:40.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T22:38:40.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:38:41.282 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T22:38:41.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T22:38:41.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:38:41.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:38:41.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:38:41.283 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:38:41.313 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:41.310+0000 7f78a2aec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:38:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:41.314+0000 7f78a2aec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:38:41.317 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:41.314+0000 7f78a2aec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:38:41.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T22:38:41.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:38:41.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:38:41.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:38:41.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:38:41.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:41.745 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:38:41.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:38:41.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:41.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:42.192 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:42.578 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:42.574+0000 7f78a2aec7c0 -1 Falling back to public interface 2024-09-15T22:38:43.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:43.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:43.194 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:38:43.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:38:43.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:43.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:43.633 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:44.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:44.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:44.635 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:38:44.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:38:44.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:44.636 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:45.057 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:45.457 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:38:45.454+0000 7f78a2aec7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T22:38:46.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:46.060 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:46.060 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:38:46.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:38:46.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:46.061 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:46.802 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:47.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:47.803 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:47.804 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:38:47.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:38:47.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:47.804 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:48.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:49.503 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:49.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:49.504 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:38:49.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:38:49.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:49.505 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:49.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:50.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:50.943 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:50.943 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:38:50.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:38:50.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:50.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:51.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:52.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:52.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:52.409 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:38:52.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:38:52.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:52.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:52.921 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:53.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:53.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:53.923 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:38:53.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:38:53.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:53.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:54.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:55.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:55.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:55.455 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:38:55.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:38:55.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:55.456 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:55.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:56.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:56.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:56.893 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:38:56.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:38:56.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:56.894 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:57.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:58.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:58.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:58.431 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:38:58.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:38:58.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:58.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:38:58.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:38:59.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:38:59.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:38:59.869 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:38:59.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:38:59.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:38:59.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:39:00.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:01.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:01.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:01.444 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:39:01.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:39:01.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:01.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:39:02.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:03.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:03.183 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:03.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:39:03.184 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:39:03.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:03.184 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:39:03.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:04.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:04.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:04.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:39:04.988 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:39:04.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:04.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:39:05.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:06.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:06.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:06.788 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:39:06.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:39:06.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:06.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:39:07.863 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:08.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:08.865 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:08.865 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:39:08.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:39:08.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:08.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:39:09.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:10.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:10.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:10.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:39:10.761 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:39:10.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:10.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:39:11.307 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/547597873,v1:127.0.0.1:6827/547597873] [v2:127.0.0.1:6828/547597873,v1:127.0.0.1:6829/547597873] exists,up 4d033fea-3ac4-4865-bfa1-1c7aa5cf1680 2024-09-15T22:39:11.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:39:11.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:39:11.308 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:39:11.309 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:39:11.309 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:39:11.309 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:39:11.310 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:39:11.310 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:11.310 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:11.310 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:39:11.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:39:11.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:39:11.421 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:39:11.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:39:11.438 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:36:52.923+0000 7f86156077c0 0 load: jerasure load: lrc 2024-09-15T22:39:11.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_subread_missing: local poolname=pool-jerasure 2024-09-15T22:39:11.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_subread_missing: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T22:39:11.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:39:11.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:39:11.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T22:39:11.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:39:11.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T22:39:11.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:39:11.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T22:39:12.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:39:12.178 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:39:13.348 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:39:13.365 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:39:14.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:39:14.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:39:14.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:39:14.367 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:39:14.368 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:39:14.369 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:39:14.369 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:39:14.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:39:14.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:39:14.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:39:14.498 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:39:14.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:39:14.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:39:14.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:39:14.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:39:14.499 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:39:14.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:39:14.939 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:39:14.939 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:39:14.939 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:39:14.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:39:14.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:14.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:39:15.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803803 2024-09-15T22:39:15.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803803 2024-09-15T22:39:15.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803803' 2024-09-15T22:39:15.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:15.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:39:15.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574871 2024-09-15T22:39:15.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574871 2024-09-15T22:39:15.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803803 1-55834574871' 2024-09-15T22:39:15.282 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:15.282 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:39:15.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378640 2024-09-15T22:39:15.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378640 2024-09-15T22:39:15.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803803 1-55834574871 2-81604378640' 2024-09-15T22:39:15.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:15.430 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:39:15.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084296 2024-09-15T22:39:15.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084296 2024-09-15T22:39:15.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803803 1-55834574871 2-81604378640 3-120259084296' 2024-09-15T22:39:15.577 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:15.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803803 2024-09-15T22:39:15.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:15.580 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:39:15.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803803 2024-09-15T22:39:15.581 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:15.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803803 2024-09-15T22:39:15.582 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803803' 2024-09-15T22:39:15.582 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803803 2024-09-15T22:39:15.583 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:39:16.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803801 -lt 25769803803 2024-09-15T22:39:16.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:39:17.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:39:17.043 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:39:17.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803804 -lt 25769803803 2024-09-15T22:39:17.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:17.512 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574871 2024-09-15T22:39:17.512 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:17.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:39:17.515 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:17.515 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574871 2024-09-15T22:39:17.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574871 2024-09-15T22:39:17.516 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574871 2024-09-15T22:39:17.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574871' 2024-09-15T22:39:17.517 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:39:17.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574872 -lt 55834574871 2024-09-15T22:39:17.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:17.971 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378640 2024-09-15T22:39:17.971 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:17.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:39:17.974 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378640 2024-09-15T22:39:17.974 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:17.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378640 2024-09-15T22:39:17.975 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 81604378640 2024-09-15T22:39:17.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378640' 2024-09-15T22:39:17.976 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:39:18.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378641 -lt 81604378640 2024-09-15T22:39:18.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:18.440 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:18.440 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084296 2024-09-15T22:39:18.441 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:39:18.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:18.442 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084296 2024-09-15T22:39:18.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084296 2024-09-15T22:39:18.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084296' 2024-09-15T22:39:18.443 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 120259084296 2024-09-15T22:39:18.444 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:39:18.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084297 -lt 120259084296 2024-09-15T22:39:18.908 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:39:18.908 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:39:18.908 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:39:19.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:39:19.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:39:19.472 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:39:19.472 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:39:19.472 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:39:19.472 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:39:19.473 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:39:19.473 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:39:19.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:39:19.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:39:19.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:39:19.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:39:20.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:39:20.472 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_subread_missing: local shard_id=1 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_subread_missing: rados_put_get_data remove td/test-erasure-eio 1 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=remove 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-remove-81867-1 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-remove-81867-1 2024-09-15T22:39:20.473 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:39:20.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:39:20.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-remove-81867-1 2024-09-15T22:39:20.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:39:20.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:39:20.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:39:20.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:39:20.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:39:20.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:39:20.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:39:20.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:39:20.474 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-remove-81867-1 td/test-erasure-eio/ORIGINAL 2024-09-15T22:39:20.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-81867-1 td/test-erasure-eio 1 2024-09-15T22:39:20.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-15T22:39:20.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-15T22:39:20.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-15T22:39:20.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-15T22:39:20.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-15T22:39:20.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-15T22:39:20.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-81867-1 2024-09-15T22:39:20.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-15T22:39:20.626 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-15T22:39:20.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-15T22:39:20.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=1 2024-09-15T22:39:20.627 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-15T22:39:20.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-81867-1 2024-09-15T22:39:20.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:39:20.627 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-remove-81867-1 2024-09-15T22:39:20.627 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:39:20.627 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-remove-81867-1 2024-09-15T22:39:21.105 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=1 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 1 obj-remove-81867-1 remove 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:39:21.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-remove-81867-1 remove 2024-09-15T22:39:21.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:39:21.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:39:21.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T22:39:21.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:39:21.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T22:39:21.107 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:39:21.107 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:39:21.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:39:21.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:39:21.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:39:21.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:39:21.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-remove-81867-1 remove 2024-09-15T22:39:21.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:39:21.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:39:21.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T22:39:21.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:39:21.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T22:39:21.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-remove-81867-1 remove 2024-09-15T22:39:23.442 INFO:tasks.workunit.client.0.smithi104.stdout:remove 1#2:3c6705b3:::obj-remove-81867-1:head# 2024-09-15T22:39:23.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:39:23.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:39:23.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:39:23.984 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:39:23.984 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:39:23.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:39:23.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:39:23.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:39:23.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T22:39:23.986 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.1 2024-09-15T22:39:23.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:39:23.987 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T22:39:23.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T22:39:23.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:39:23.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:39:23.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:39:24.020 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:24.018+0000 7f744270d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:39:24.040 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:24.038+0000 7f744270d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:39:24.042 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:24.038+0000 7f744270d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:39:24.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T22:39:24.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:39:24.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:39:24.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:39:24.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:39:24.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:24.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:39:24.430 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:39:24.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:24.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:39:24.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:25.562 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:25.557+0000 7f744270d7c0 -1 Falling back to public interface 2024-09-15T22:39:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:39:25.941 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:39:25.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:25.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:39:26.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:27.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:27.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:27.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:39:27.433 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:39:27.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:27.433 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:39:27.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:28.577 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:28.573+0000 7f744270d7c0 -1 osd.1 36 log_to_monitors true 2024-09-15T22:39:28.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:28.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:28.882 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:39:28.882 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:39:28.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:28.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:39:29.667 INFO:tasks.workunit.client.0.smithi104.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 37 last_clean_interval [13,36) [v2:127.0.0.1:6810/792361840,v1:127.0.0.1:6811/792361840] [v2:127.0.0.1:6812/792361840,v1:127.0.0.1:6813/792361840] exists,up 9e0f8581-4b76-4b0d-93d0-b170ba1a5bbc 2024-09-15T22:39:29.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:39:29.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:39:29.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:39:29.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:39:29.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:39:29.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:39:29.667 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:39:29.668 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:39:29.668 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:39:29.668 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:39:29.668 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:39:29.668 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:39:29.668 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:39:29.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:39:29.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:39:29.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:39:29.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:39:29.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:39:29.785 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:39:30.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:39:30.403 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:39:30.403 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:39:30.403 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:39:30.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:39:30.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:30.404 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:39:30.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803809 2024-09-15T22:39:30.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803809 2024-09-15T22:39:30.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809' 2024-09-15T22:39:30.598 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:30.599 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:39:30.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659139 2024-09-15T22:39:30.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659139 2024-09-15T22:39:30.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-176093659139' 2024-09-15T22:39:30.772 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:30.773 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:39:30.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378645 2024-09-15T22:39:30.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378645 2024-09-15T22:39:30.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-176093659139 2-81604378645' 2024-09-15T22:39:30.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:30.928 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:39:31.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084302 2024-09-15T22:39:31.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084302 2024-09-15T22:39:31.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-176093659139 2-81604378645 3-120259084302' 2024-09-15T22:39:31.074 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:31.074 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803809 2024-09-15T22:39:31.074 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:31.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:39:31.076 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803809 2024-09-15T22:39:31.076 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:31.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803809 2024-09-15T22:39:31.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-09-15T22:39:31.077 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803809 2024-09-15T22:39:31.077 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:39:31.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803806 -lt 25769803809 2024-09-15T22:39:31.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:39:32.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:39:32.515 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:39:32.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803809 -lt 25769803809 2024-09-15T22:39:32.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:32.977 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-176093659139 2024-09-15T22:39:32.978 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:32.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:39:32.981 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-176093659139 2024-09-15T22:39:32.981 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:32.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659139 2024-09-15T22:39:32.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 176093659139' 2024-09-15T22:39:32.983 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 176093659139 2024-09-15T22:39:32.983 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:39:33.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659139 -lt 176093659139 2024-09-15T22:39:33.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:33.415 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378645 2024-09-15T22:39:33.415 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:33.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:39:33.417 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378645 2024-09-15T22:39:33.417 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:33.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378645 2024-09-15T22:39:33.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378645' 2024-09-15T22:39:33.418 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 81604378645 2024-09-15T22:39:33.419 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:39:33.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378645 -lt 81604378645 2024-09-15T22:39:33.841 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:33.842 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084302 2024-09-15T22:39:33.842 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:33.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:39:33.845 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084302 2024-09-15T22:39:33.845 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:33.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084302 2024-09-15T22:39:33.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084302' 2024-09-15T22:39:33.847 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 120259084302 2024-09-15T22:39:33.847 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:39:34.281 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084302 -lt 120259084302 2024-09-15T22:39:34.282 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:39:34.282 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:39:34.282 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:39:34.855 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:39:34.856 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:39:34.856 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:39:34.857 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:39:34.857 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:39:34.857 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:39:34.857 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:39:34.857 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:39:35.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T22:39:35.261 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:39:35.261 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:39:35.262 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:39:35.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T22:39:35.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T22:39:35.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T22:39:35.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T22:39:35.796 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T22:39:35.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T22:39:35.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T22:39:35.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:39:35.898 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:39:35.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:39:35.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:39:35.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:39:35.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:39:35.899 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:39:36.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:39:36.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:39:36.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:39:36.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:39:36.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:39:36.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:39:36.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:39:36.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-81867-1 2024-09-15T22:39:36.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:39:36.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:39:36.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-81867-1 2024-09-15T22:39:36.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:39:36.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:39:36.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-81867-1 td/test-erasure-eio/COPY 2024-09-15T22:39:37.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:39:37.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:39:37.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-15T22:39:37.044 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-15T22:39:37.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=2 2024-09-15T22:39:37.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-81867-1 td/test-erasure-eio 2 2024-09-15T22:39:37.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-15T22:39:37.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-15T22:39:37.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-15T22:39:37.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-15T22:39:37.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-15T22:39:37.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-15T22:39:37.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-81867-1 2024-09-15T22:39:37.048 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-15T22:39:37.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-15T22:39:37.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-15T22:39:37.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=2 2024-09-15T22:39:37.049 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-15T22:39:37.049 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-81867-1 2024-09-15T22:39:37.049 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:39:37.049 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-remove-81867-1 2024-09-15T22:39:37.049 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:39:37.050 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-remove-81867-1 2024-09-15T22:39:37.532 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:39:37.532 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:39:37.532 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:39:37.532 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:39:37.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2024-09-15T22:39:37.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-15T22:39:37.532 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=2 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 2 obj-remove-81867-1 remove 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-remove-81867-1 remove 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:39:37.533 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:39:37.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:39:37.534 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:39:37.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:39:37.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-remove-81867-1 remove 2024-09-15T22:39:37.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T22:39:37.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T22:39:37.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T22:39:37.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T22:39:37.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-15T22:39:37.843 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-remove-81867-1 remove 2024-09-15T22:39:40.088 INFO:tasks.workunit.client.0.smithi104.stdout:remove 2#2:3c6705b3:::obj-remove-81867-1:head# 2024-09-15T22:39:40.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-15T22:39:40.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:39:40.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:39:40.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T22:39:40.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:39:40.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:39:40.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:39:40.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:39:40.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:39:40.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:39:40.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:39:40.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:39:40.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:39:40.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:39:40.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:39:40.640 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:39:40.640 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:39:40.640 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:39:40.641 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:39:40.642 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:39:40.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:39:40.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T22:39:40.643 INFO:tasks.workunit.client.0.smithi104.stderr:start osd.2 2024-09-15T22:39:40.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:39:40.643 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T22:39:40.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T22:39:40.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:39:40.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:39:40.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:39:40.681 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:40.677+0000 7f45c47da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:39:40.684 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:40.681+0000 7f45c47da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:39:40.685 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:40.681+0000 7f45c47da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:39:41.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T22:39:41.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:39:41.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:39:41.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:39:41.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:39:41.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:41.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:39:41.128 INFO:tasks.workunit.client.0.smithi104.stderr:0 2024-09-15T22:39:41.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:41.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:39:41.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:41.922 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:41.917+0000 7f45c47da7c0 -1 Falling back to public interface 2024-09-15T22:39:42.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:42.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:42.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:39:42.549 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:39:42.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:42.550 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:39:42.980 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:43.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:43.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:43.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:39:43.982 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:39:43.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:43.983 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:39:44.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:39:44.795 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:44.789+0000 7f45c47da7c0 -1 osd.2 43 log_to_monitors true 2024-09-15T22:39:45.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:39:45.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:39:45.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:39:45.413 INFO:tasks.workunit.client.0.smithi104.stderr:3 2024-09-15T22:39:45.413 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:39:45.414 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:39:46.135 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 up in weight 1 up_from 48 up_thru 37 down_at 44 last_clean_interval [19,43) [v2:127.0.0.1:6818/1443079088,v1:127.0.0.1:6819/1443079088] [v2:127.0.0.1:6820/1443079088,v1:127.0.0.1:6821/1443079088] exists,up 94ccd8ff-6423-4e72-98c5-8f82d4412245 2024-09-15T22:39:46.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:39:46.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:39:46.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:39:46.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T22:39:46.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:39:46.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:39:46.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:39:46.136 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:39:46.137 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:39:46.137 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:39:46.137 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:39:46.137 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:39:46.137 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:39:46.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:39:46.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:39:46.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:39:46.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:39:46.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:39:46.259 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:39:46.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:39:46.942 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:39:46.942 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:39:46.942 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:39:46.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:39:46.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:46.943 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:39:47.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803814 2024-09-15T22:39:47.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803814 2024-09-15T22:39:47.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814' 2024-09-15T22:39:47.115 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:47.115 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:39:47.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659144 2024-09-15T22:39:47.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659144 2024-09-15T22:39:47.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814 1-176093659144' 2024-09-15T22:39:47.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:47.266 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:39:47.425 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430211 2024-09-15T22:39:47.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430211 2024-09-15T22:39:47.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814 1-176093659144 2-206158430211' 2024-09-15T22:39:47.426 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:39:47.426 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:39:47.584 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084307 2024-09-15T22:39:47.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084307 2024-09-15T22:39:47.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814 1-176093659144 2-206158430211 3-120259084307' 2024-09-15T22:39:47.585 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:47.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803814 2024-09-15T22:39:47.585 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:47.587 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:39:47.587 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803814 2024-09-15T22:39:47.588 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:47.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803814 2024-09-15T22:39:47.589 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803814' 2024-09-15T22:39:47.589 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.0 seq 25769803814 2024-09-15T22:39:47.589 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:39:48.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803812 -lt 25769803814 2024-09-15T22:39:48.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:39:49.014 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:39:49.015 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:39:49.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803814 -lt 25769803814 2024-09-15T22:39:49.499 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:49.500 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-176093659144 2024-09-15T22:39:49.500 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:49.502 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:39:49.503 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-176093659144 2024-09-15T22:39:49.503 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:49.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659144 2024-09-15T22:39:49.504 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 176093659144' 2024-09-15T22:39:49.504 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.1 seq 176093659144 2024-09-15T22:39:49.505 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:39:49.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659144 -lt 176093659144 2024-09-15T22:39:49.994 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:49.994 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-206158430211 2024-09-15T22:39:49.995 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:49.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:39:49.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-206158430211 2024-09-15T22:39:49.997 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:49.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430211 2024-09-15T22:39:49.998 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 206158430211' 2024-09-15T22:39:49.998 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.2 seq 206158430211 2024-09-15T22:39:49.999 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:39:50.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430211 -lt 206158430211 2024-09-15T22:39:50.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:39:50.406 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084307 2024-09-15T22:39:50.406 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:39:50.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:39:50.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084307 2024-09-15T22:39:50.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:39:50.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084307 2024-09-15T22:39:50.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084307' 2024-09-15T22:39:50.412 INFO:tasks.workunit.client.0.smithi104.stderr:waiting osd.3 seq 120259084307 2024-09-15T22:39:50.412 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:39:50.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084307 -lt 120259084307 2024-09-15T22:39:50.833 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:39:50.834 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:39:50.834 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:39:51.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:39:51.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:39:51.386 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:39:51.386 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:39:51.386 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:39:51.386 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:39:51.387 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:39:51.387 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:39:51.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:39:51.815 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:39:51.816 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:39:51.816 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:39:52.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:39:52.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:39:52.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:39:52.359 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-15T22:39:52.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:188: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-81867-1 2024-09-15T22:39:52.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:39:52.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:39:52.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-81867-1 2024-09-15T22:39:52.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:39:52.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:39:52.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-81867-1 td/test-erasure-eio/COPY 2024-09-15T22:39:52.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:39:52.401 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:39:52.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-15T22:39:52.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_subread_missing: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:39:52.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:39:52.403 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:39:52.890 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:39:52.907 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:39:53.524 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:39:53.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:39:53.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:39:53.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:39:53.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:39:53.542 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:39:53.542 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:39:53.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:39:53.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:39:53.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:39:53.653 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:39:53.654 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:39:53.655 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:39:53.655 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:39:53.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:39:53.656 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:39:53.657 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:39:53.657 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:39:53.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:39:53.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:39:53.658 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:39:53.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:39:53.659 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:39:53.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:39:53.680 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:39:53.680 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:53.680 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:53.680 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:39:53.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:39:53.682 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:39:53.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:39:53.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:39:53.684 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:39:53.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:39:53.685 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:39:53.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:39:53.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:39:53.687 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:39:53.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:39:53.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:39:53.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:39:53.688 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:39:53.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:39:53.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:39:53.689 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:39:53.690 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:39:53.691 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:53.691 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:53.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:39:53.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:39:53.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:39:53.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T22:39:53.693 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:39:53.693 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:53.693 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:53.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T22:39:53.695 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:39:53.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:39:53.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:39:53.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T22:39:53.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T22:39:53.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T22:39:53.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:39:53.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:39:53.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:39:53.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T22:39:53.696 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T22:39:54.177 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:39:54.177 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:39:54.178 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:39:54.178 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:39:54.178 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:54.178 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:54.179 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:39:54.179 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T22:39:54.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:39:54.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:39:54.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:39:54.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:39:54.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:39:54.374 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:39:54.375 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:39:54.375 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:39:54.375 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:39:54.375 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:39:54.375 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:54.375 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:54.375 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:39:54.376 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:39:54.376 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T22:39:54.497 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:39:54.497 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:39:54.498 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:39:54.498 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:39:54.498 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:39:54.498 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:39:54.498 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:39:54.498 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:39:54.498 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:39:54.498 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:54.498 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:54.499 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:39:54.499 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:39:54.499 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T22:39:54.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T22:39:54.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T22:39:54.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:39:54.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:39:54.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:39:54.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T22:39:54.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:39:55.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:39:55.511 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:39:55.511 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:39:55.511 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:39:55.511 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:55.511 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:55.511 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:39:55.512 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:39:55.512 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:39:55.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T22:39:55.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T22:39:55.833 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T22:39:55.849 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:39:56.852 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T22:39:56.852 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:39:56.852 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:39:56.853 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:39:56.853 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:56.853 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:56.853 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:39:56.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T22:39:56.854 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T22:39:56.963 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:39:56.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T22:39:56.977 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:39:54.217+0000 7f38407a5e00 0 load: jerasure load: lrc 2024-09-15T22:39:56.977 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_0 td/test-erasure-eio 2024-09-15T22:39:56.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:350: TEST_rados_get_with_subreadall_eio_shard_0: local dir=td/test-erasure-eio 2024-09-15T22:39:56.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:351: TEST_rados_get_with_subreadall_eio_shard_0: local shard_id=0 2024-09-15T22:39:56.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:353: TEST_rados_get_with_subreadall_eio_shard_0: setup_osds 4 2024-09-15T22:39:56.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T22:39:56.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T22:39:56.979 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T22:39:56.980 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T22:39:56.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:39:56.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T22:39:56.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:39:56.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:39:56.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:39:56.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:39:56.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T22:39:56.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:39:56.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:39:56.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:39:56.986 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T22:39:56.987 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:39:56.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8d3c94f5-8c12-4fdf-8d6b-c90f4b608b6d 2024-09-15T22:39:56.988 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 8d3c94f5-8c12-4fdf-8d6b-c90f4b608b6d 2024-09-15T22:39:56.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8d3c94f5-8c12-4fdf-8d6b-c90f4b608b6d' 2024-09-15T22:39:56.989 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:39:57.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA9YudmTSRIABAAuFfuHZpnbEoFhUoRklWBvw== 2024-09-15T22:39:57.005 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA9YudmTSRIABAAuFfuHZpnbEoFhUoRklWBvw=="}' 2024-09-15T22:39:57.006 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8d3c94f5-8c12-4fdf-8d6b-c90f4b608b6d -i td/test-erasure-eio/0/new.json 2024-09-15T22:39:57.293 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:39:57.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T22:39:57.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9YudmTSRIABAAuFfuHZpnbEoFhUoRklWBvw== --osd-uuid 8d3c94f5-8c12-4fdf-8d6b-c90f4b608b6d 2024-09-15T22:39:57.342 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:57.337+0000 7f753fbfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:39:57.361 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:57.357+0000 7f753fbfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:39:57.362 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:57.357+0000 7f753fbfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:39:57.363 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:57.357+0000 7f753fbfa7c0 -1 bdev(0x55e24551bc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:39:57.363 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:39:57.361+0000 7f753fbfa7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T22:40:03.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T22:40:03.054 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:40:03.055 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T22:40:03.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:40:03.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:40:03.616 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T22:40:03.616 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:40:03.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:40:03.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:40:03.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:40:03.617 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:40:03.649 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:03.645+0000 7fc38d9f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:40:03.664 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:03.661+0000 7fc38d9f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:40:03.665 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:03.661+0000 7fc38d9f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:40:04.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:40:04.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:40:04.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:40:04.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:40:04.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:40:04.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:04.095 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:40:04.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:40:04.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:04.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:04.539 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:05.198 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:05.193+0000 7fc38d9f27c0 -1 Falling back to public interface 2024-09-15T22:40:05.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:05.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:05.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:40:05.541 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:40:05.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:05.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:06.021 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:07.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:07.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:07.024 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:40:07.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:40:07.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:07.024 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:07.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:08.099 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:08.093+0000 7fc38d9f27c0 -1 osd.0 0 log_to_monitors true 2024-09-15T22:40:08.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:08.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:08.549 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:40:08.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:40:08.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:08.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:09.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:10.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:10.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:10.146 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:40:10.146 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:40:10.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:10.147 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:10.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:11.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:11.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:11.607 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:40:11.607 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:40:11.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:11.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:12.194 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:13.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:13.195 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:13.196 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:40:13.196 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:40:13.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:13.197 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:13.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:14.640 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:14.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:14.641 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:40:14.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:40:14.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:14.641 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:15.064 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:16.065 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:16.066 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:40:16.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:40:16.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:16.066 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:16.539 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:17.540 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:17.541 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:40:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:40:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:17.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:18.026 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:19.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:19.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:19.029 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:40:19.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:40:19.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:19.030 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:19.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:20.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:20.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:20.467 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:40:20.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:40:20.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:20.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:21.301 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:22.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:22.303 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:22.304 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:40:22.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:40:22.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:22.304 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:23.080 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:24.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:24.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:24.082 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:40:24.083 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:40:24.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:24.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:25.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:26.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:26.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:26.209 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:40:26.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:40:26.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:26.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:26.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:27.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:27.975 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:27.975 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:40:27.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:40:27.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:27.976 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:28.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:29.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:29.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:29.789 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:40:29.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:40:29.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:29.789 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:30.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:31.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:31.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:31.545 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:40:31.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:40:31.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:31.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:40:31.983 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3145045342,v1:127.0.0.1:6803/3145045342] [v2:127.0.0.1:6804/3145045342,v1:127.0.0.1:6805/3145045342] exists,up 8d3c94f5-8c12-4fdf-8d6b-c90f4b608b6d 2024-09-15T22:40:31.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:40:31.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:40:31.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:40:31.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:40:31.985 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:40:31.986 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:40:31.987 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:40:31.987 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:40:31.987 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:40:31.987 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:40:31.987 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:40:31.987 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:40:31.987 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:40:31.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:40:31.988 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:40:31.989 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:40:31.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c6f5edf5-b536-4d61-b8a4-39079f7d7d26 2024-09-15T22:40:31.990 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 c6f5edf5-b536-4d61-b8a4-39079f7d7d26' 2024-09-15T22:40:31.990 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 c6f5edf5-b536-4d61-b8a4-39079f7d7d26 2024-09-15T22:40:31.990 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:40:32.011 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBgYudmDYCkABAA6DK/x5ks2IOdV9nMTAdSGA== 2024-09-15T22:40:32.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBgYudmDYCkABAA6DK/x5ks2IOdV9nMTAdSGA=="}' 2024-09-15T22:40:32.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c6f5edf5-b536-4d61-b8a4-39079f7d7d26 -i td/test-erasure-eio/1/new.json 2024-09-15T22:40:32.848 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:40:32.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T22:40:32.867 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBgYudmDYCkABAA6DK/x5ks2IOdV9nMTAdSGA== --osd-uuid c6f5edf5-b536-4d61-b8a4-39079f7d7d26 2024-09-15T22:40:32.901 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:32.897+0000 7f9c32cbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:40:32.903 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:32.897+0000 7f9c32cbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:40:32.904 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:32.901+0000 7f9c32cbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:40:32.904 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:32.901+0000 7f9c32cbd7c0 -1 bdev(0x562bf65f1c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:40:32.905 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:32.901+0000 7f9c32cbd7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T22:40:39.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T22:40:39.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:40:39.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:40:39.155 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T22:40:39.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:40:39.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:40:39.717 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T22:40:39.717 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:40:39.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:40:39.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:40:39.718 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:40:39.750 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:39.745+0000 7f8d518a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:40:39.767 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:39.761+0000 7f8d518a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:40:39.769 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:39.765+0000 7f8d518a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:40:40.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:40:40.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:40:40.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:40:40.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:40:40.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:40:40.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:40.180 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:40:40.181 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:40:40.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:40.181 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:40.683 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:41.038 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:41.033+0000 7f8d518a17c0 -1 Falling back to public interface 2024-09-15T22:40:41.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:41.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:41.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:40:41.685 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:40:41.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:41.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:42.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:43.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:43.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:43.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:40:43.103 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:40:43.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:43.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:43.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:43.851 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:40:43.845+0000 7f8d518a17c0 -1 osd.1 0 log_to_monitors true 2024-09-15T22:40:44.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:44.541 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:44.541 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:40:44.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:40:44.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:44.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:45.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:46.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:46.106 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:46.106 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:40:46.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:40:46.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:46.107 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:46.758 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:47.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:47.760 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:47.761 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:40:47.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:40:47.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:47.761 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:48.203 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:49.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:49.205 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:40:49.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:49.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:40:49.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:49.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:49.684 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:50.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:50.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:50.686 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:40:50.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:40:50.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:50.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:51.322 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:52.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:52.324 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:52.324 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:40:52.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:40:52.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:52.325 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:52.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:53.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:53.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:53.834 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:40:53.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:40:53.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:53.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:54.264 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:55.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:55.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:55.266 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:40:55.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:40:55.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:55.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:55.829 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:56.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:56.831 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:56.832 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:40:56.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:40:56.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:56.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:57.672 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:40:58.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:40:58.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:40:58.675 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:40:58.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:40:58.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:40:58.675 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:40:59.647 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:00.648 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:00.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:00.649 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:41:00.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:41:00.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:00.649 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:41:01.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:02.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:02.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:02.729 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:41:02.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:41:02.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:02.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:41:03.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:04.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:04.813 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:04.814 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:41:04.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:41:04.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:04.814 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:41:05.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:06.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:06.410 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:41:06.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:06.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:41:06.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:06.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:41:06.877 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:07.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:07.879 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:07.879 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:41:07.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:41:07.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:07.880 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:41:08.390 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4049295873,v1:127.0.0.1:6811/4049295873] [v2:127.0.0.1:6812/4049295873,v1:127.0.0.1:6813/4049295873] exists,up c6f5edf5-b536-4d61-b8a4-39079f7d7d26 2024-09-15T22:41:08.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:41:08.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:41:08.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:41:08.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:41:08.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T22:41:08.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:41:08.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:41:08.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:41:08.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:41:08.392 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:41:08.393 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:41:08.394 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:41:08.395 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:41:08.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:41:08.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:41:08.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:41:08.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:41:08.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:41:08.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:41:08.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:41:08.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:41:08.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:41:08.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:41:08.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:41:08.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:41:08.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:41:08.396 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:41:08.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=25c62bc0-9ca4-4b40-9539-7575bb06ae8f 2024-09-15T22:41:08.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 25c62bc0-9ca4-4b40-9539-7575bb06ae8f' 2024-09-15T22:41:08.397 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 25c62bc0-9ca4-4b40-9539-7575bb06ae8f 2024-09-15T22:41:08.397 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:41:08.417 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCEYudmCY7cGBAAuIzZz4UaIMOY39gCRkFeRg== 2024-09-15T22:41:08.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCEYudmCY7cGBAAuIzZz4UaIMOY39gCRkFeRg=="}' 2024-09-15T22:41:08.418 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 25c62bc0-9ca4-4b40-9539-7575bb06ae8f -i td/test-erasure-eio/2/new.json 2024-09-15T22:41:09.070 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:41:09.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T22:41:09.088 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEYudmCY7cGBAAuIzZz4UaIMOY39gCRkFeRg== --osd-uuid 25c62bc0-9ca4-4b40-9539-7575bb06ae8f 2024-09-15T22:41:09.115 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:09.108+0000 7f05cc4847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:09.117 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:09.112+0000 7f05cc4847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:09.118 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:09.112+0000 7f05cc4847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:09.118 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:09.112+0000 7f05cc4847c0 -1 bdev(0x55a326501c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:41:09.118 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:09.112+0000 7f05cc4847c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T22:41:15.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T22:41:15.753 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:41:15.755 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T22:41:15.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:41:15.755 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:41:16.340 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:41:16.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:41:16.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:41:16.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:41:16.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:41:16.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:41:16.373 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:16.368+0000 7fb8e1ee97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:16.380 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:16.376+0000 7fb8e1ee97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:16.381 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:16.376+0000 7fb8e1ee97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:16.782 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:41:16.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:41:16.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:41:16.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:41:16.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:41:16.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:16.783 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:41:16.783 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:41:16.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:16.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:17.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:17.614 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:17.608+0000 7fb8e1ee97c0 -1 Falling back to public interface 2024-09-15T22:41:18.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:18.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:18.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:41:18.272 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:41:18.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:18.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:18.721 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:19.722 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:19.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:19.723 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:41:19.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:41:19.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:19.724 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:20.160 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:20.481 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:20.476+0000 7fb8e1ee97c0 -1 osd.2 0 log_to_monitors true 2024-09-15T22:41:21.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:21.161 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:21.162 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:41:21.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:41:21.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:21.162 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:21.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:22.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:22.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:22.729 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:41:22.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:41:22.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:22.730 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:23.266 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:24.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:24.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:24.268 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:41:24.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:41:24.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:24.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:24.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:25.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:25.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:25.912 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:41:25.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:41:25.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:25.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:26.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:27.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:27.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:27.412 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:41:27.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:41:27.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:27.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:27.844 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:28.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:28.846 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:28.846 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:41:28.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:41:28.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:28.847 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:29.452 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:30.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:30.454 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:30.454 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:41:30.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:41:30.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:30.455 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:30.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:31.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:31.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:31.967 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:41:31.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:41:31.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:31.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:32.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:33.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:33.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:33.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:41:33.517 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:41:33.517 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:33.518 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:34.017 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:35.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:35.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:35.019 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:41:35.019 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:41:35.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:35.020 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:36.205 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:37.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:37.207 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:37.208 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:41:37.208 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:41:37.208 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:37.208 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:37.644 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:38.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:38.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:38.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:41:38.646 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:41:38.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:38.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:39.709 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:40.710 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:40.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:40.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:41:40.711 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:41:40.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:40.712 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:41.606 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:42.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:42.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:42.608 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:41:42.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:41:42.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:42.609 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:43.479 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:44.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:44.481 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:44.481 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:41:44.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:41:44.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:44.482 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:41:44.946 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2895117286,v1:127.0.0.1:6819/2895117286] [v2:127.0.0.1:6820/2895117286,v1:127.0.0.1:6821/2895117286] exists,up 25c62bc0-9ca4-4b40-9539-7575bb06ae8f 2024-09-15T22:41:44.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:41:44.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:41:44.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:41:44.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:41:44.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T22:41:44.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:41:44.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:41:44.948 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:41:44.949 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:41:44.950 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:41:44.950 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:41:44.950 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:41:44.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:41:44.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:41:44.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:41:44.953 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:41:44.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=53575305-0bb4-4297-90d0-7d72d0886dc3 2024-09-15T22:41:44.953 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 53575305-0bb4-4297-90d0-7d72d0886dc3' 2024-09-15T22:41:44.953 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 53575305-0bb4-4297-90d0-7d72d0886dc3 2024-09-15T22:41:44.953 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:41:44.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCoYudmqLL8ORAAV+v9oj4Cm6HAGtnVIegiDQ== 2024-09-15T22:41:44.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCoYudmqLL8ORAAV+v9oj4Cm6HAGtnVIegiDQ=="}' 2024-09-15T22:41:44.974 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 53575305-0bb4-4297-90d0-7d72d0886dc3 -i td/test-erasure-eio/3/new.json 2024-09-15T22:41:45.820 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:41:45.837 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T22:41:45.838 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoYudmqLL8ORAAV+v9oj4Cm6HAGtnVIegiDQ== --osd-uuid 53575305-0bb4-4297-90d0-7d72d0886dc3 2024-09-15T22:41:45.873 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:45.868+0000 7f93b7c497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:45.950 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:45.944+0000 7f93b7c497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:45.952 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:45.948+0000 7f93b7c497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:45.952 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:45.948+0000 7f93b7c497c0 -1 bdev(0x55cba84efc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T22:41:45.953 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:45.948+0000 7f93b7c497c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T22:41:52.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T22:41:52.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:41:52.126 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T22:41:52.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T22:41:52.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:41:52.686 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T22:41:52.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T22:41:52.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:41:52.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:41:52.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:41:52.687 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:41:52.714 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:52.708+0000 7f86898ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:52.737 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:52.732+0000 7f86898ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:52.738 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:52.732+0000 7f86898ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:41:53.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T22:41:53.103 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:41:53.104 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:41:53.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:41:53.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:41:53.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:41:53.104 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:53.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:41:53.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:41:53.105 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:53.592 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:53.782 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:53.776+0000 7f86898ac7c0 -1 Falling back to public interface 2024-09-15T22:41:54.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:54.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:54.594 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:41:54.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:41:54.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:54.595 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:41:55.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:56.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:56.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:56.037 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:41:56.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:41:56.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:56.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:41:56.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:56.604 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:41:56.600+0000 7f86898ac7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T22:41:57.493 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:57.494 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:57.494 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:41:57.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:41:57.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:57.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:41:58.307 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:41:59.309 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:41:59.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:41:59.310 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:41:59.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:41:59.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:41:59.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:41:59.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:00.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:00.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:00.956 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:42:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:42:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:00.957 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:01.457 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:02.458 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:02.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:02.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:42:02.459 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:42:02.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:02.459 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:02.923 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:03.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:03.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:03.925 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:42:03.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:42:03.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:03.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:04.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:05.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:05.436 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:05.437 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:42:05.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:42:05.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:05.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:05.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:06.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:06.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:06.929 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:42:06.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:42:06.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:06.930 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:07.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:08.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:08.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:08.430 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:42:08.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:42:08.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:08.431 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:08.979 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:09.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:09.981 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:09.981 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:42:09.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:42:09.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:09.982 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:10.608 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:11.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:11.611 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:11.611 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:42:11.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:42:11.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:11.612 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:12.260 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:13.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:13.262 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:13.262 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:42:13.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:42:13.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:13.263 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:14.292 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:15.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:15.294 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:15.294 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:42:15.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:42:15.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:15.295 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:16.223 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:17.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:17.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:17.225 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:42:17.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:42:17.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:17.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:17.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:18.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:18.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:18.970 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:42:18.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:42:18.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:18.971 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:20.252 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:21.253 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:21.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:21.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:42:21.254 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:42:21.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:21.254 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:42:21.783 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1876060634,v1:127.0.0.1:6827/1876060634] [v2:127.0.0.1:6828/1876060634,v1:127.0.0.1:6829/1876060634] exists,up 53575305-0bb4-4297-90d0-7d72d0886dc3 2024-09-15T22:42:21.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:42:21.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:42:21.784 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:42:21.784 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:42:21.785 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:42:21.785 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:42:21.785 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:42:21.785 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:21.786 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:21.786 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:42:21.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:42:21.786 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:42:21.898 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:42:21.911 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:42:21.915 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:40:05.205+0000 7fc38d9f27c0 0 load: jerasure load: lrc 2024-09-15T22:42:21.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:355: TEST_rados_get_with_subreadall_eio_shard_0: local poolname=pool-jerasure 2024-09-15T22:42:21.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:356: TEST_rados_get_with_subreadall_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T22:42:21.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:42:21.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:42:21.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T22:42:21.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:42:21.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T22:42:21.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:42:21.917 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T22:42:22.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:42:22.886 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:42:23.991 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:42:24.009 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:42:25.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:42:25.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:42:25.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:42:25.012 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:42:25.012 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:42:25.013 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:42:25.013 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:42:25.014 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:42:25.014 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:42:25.014 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:42:25.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:42:25.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:42:25.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:42:25.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:42:25.144 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:42:25.145 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:42:25.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:42:25.622 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:42:25.622 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:42:25.622 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:42:25.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:42:25.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:42:25.622 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:42:25.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803806 2024-09-15T22:42:25.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803806 2024-09-15T22:42:25.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806' 2024-09-15T22:42:25.815 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:42:25.816 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:42:25.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574875 2024-09-15T22:42:25.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574875 2024-09-15T22:42:25.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574875' 2024-09-15T22:42:25.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:42:25.968 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:42:26.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378639 2024-09-15T22:42:26.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378639 2024-09-15T22:42:26.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574875 2-81604378639' 2024-09-15T22:42:26.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:42:26.121 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:42:26.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964116997 2024-09-15T22:42:26.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964116997 2024-09-15T22:42:26.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574875 2-81604378639 3-115964116997' 2024-09-15T22:42:26.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:42:26.274 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803806 2024-09-15T22:42:26.274 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:42:26.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:42:26.277 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803806 2024-09-15T22:42:26.277 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:42:26.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803806 2024-09-15T22:42:26.278 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803806' 2024-09-15T22:42:26.278 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803806 2024-09-15T22:42:26.278 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:42:26.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803806 -lt 25769803806 2024-09-15T22:42:26.741 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:42:26.742 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:42:26.742 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574875 2024-09-15T22:42:26.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:42:26.744 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574875 2024-09-15T22:42:26.744 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:42:26.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574875 2024-09-15T22:42:26.746 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574875 2024-09-15T22:42:26.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574875' 2024-09-15T22:42:26.746 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:42:27.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574875 -lt 55834574875 2024-09-15T22:42:27.225 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:42:27.226 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378639 2024-09-15T22:42:27.226 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:42:27.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:42:27.229 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378639 2024-09-15T22:42:27.229 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:42:27.230 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378639 2024-09-15T22:42:27.230 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 81604378639 2024-09-15T22:42:27.231 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378639' 2024-09-15T22:42:27.231 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:42:27.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378639 -lt 81604378639 2024-09-15T22:42:27.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:42:27.659 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964116997 2024-09-15T22:42:27.659 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:42:27.660 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:42:27.661 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:42:27.662 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964116997 2024-09-15T22:42:27.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964116997 2024-09-15T22:42:27.663 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 115964116997 2024-09-15T22:42:27.663 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964116997' 2024-09-15T22:42:27.663 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:42:28.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964116997 -lt 115964116997 2024-09-15T22:42:28.079 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:42:28.080 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:42:28.080 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:42:28.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:42:28.658 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:42:28.658 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:42:28.659 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:42:28.659 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:42:28.659 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:42:28.659 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:42:28.659 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:42:29.079 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:42:29.080 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:42:29.080 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:42:29.080 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:358: TEST_rados_get_with_subreadall_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 recovery 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-15T22:42:29.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-81867-0 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-81867-0 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-81867-0 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:42:29.624 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-81867-0 td/test-erasure-eio/ORIGINAL 2024-09-15T22:42:29.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81867-0 td/test-erasure-eio 0 2024-09-15T22:42:29.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:42:29.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:42:29.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:42:29.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:42:29.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:42:29.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:42:29.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-81867-0 2024-09-15T22:42:29.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:42:29.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:42:29.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:42:29.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T22:42:29.809 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:42:29.809 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-81867-0 2024-09-15T22:42:29.809 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:42:29.809 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-81867-0 2024-09-15T22:42:29.810 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-81867-0 2024-09-15T22:42:29.810 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:42:30.267 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:42:30.267 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:42:30.267 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:42:30.267 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:42:30.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T22:42:30.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:42:30.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-15T22:42:30.267 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:42:30.268 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T22:42:30.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:42:30.268 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T22:42:30.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:42:30.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-15T22:42:30.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:42:30.269 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:42:30.269 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:42:30.270 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-15T22:42:30.270 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:42:30.270 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:42:30.270 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:42:30.271 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:30.271 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:30.271 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:42:30.271 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:42:30.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:42:30.383 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:42:30.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:42:30.385 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-15T22:42:30.385 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-15T22:42:30.385 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-15T22:42:30.385 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:42:30.385 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:30.385 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:30.386 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.3.asok 2024-09-15T22:42:30.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:42:30.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-81867-0 0 2024-09-15T22:42:30.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81867-0 2024-09-15T22:42:30.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:42:30.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:42:30.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81867-0 2024-09-15T22:42:30.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:42:30.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:42:30.495 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-81867-0 td/test-erasure-eio/COPY 2024-09-15T22:42:30.536 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:42:30.537 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:42:30.538 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-15T22:42:30.538 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-81867-0 2024-09-15T22:42:30.538 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:42:30.539 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-81867-0 2024-09-15T22:42:30.539 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-81867-0 2024-09-15T22:42:30.539 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:42:30.961 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:42:30.961 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:42:30.961 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:42:30.961 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:42:30.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2024-09-15T22:42:30.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-15T22:42:30.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-15T22:42:30.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T22:42:30.962 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:42:30.962 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:42:30.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:42:30.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:42:30.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:42:31.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:42:31.272 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-15T22:42:32.399 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 is already out. 2024-09-15T22:42:32.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-81867-0 2024-09-15T22:42:32.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-15T22:42:32.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:42:32.416 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-81867-0 2024-09-15T22:42:32.417 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-81867-0 2024-09-15T22:42:32.418 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:42:33.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:42:33.108 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:42:33.108 INFO:tasks.workunit.client.0.smithi104.stderr:0' 2024-09-15T22:42:33.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 0 2024-09-15T22:42:33.108 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-15T22:42:33.865 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 is already in. 2024-09-15T22:42:33.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:42:33.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:42:33.885 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:42:33.886 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:42:33.886 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:42:33.886 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:42:33.886 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:42:33.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:42:33.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:42:33.888 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:42:33.888 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T22:42:33.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:42:33.889 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T22:42:33.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T22:42:33.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:42:33.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:42:33.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:42:33.920 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:33.919+0000 7fbd15b877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:42:33.923 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:33.919+0000 7fbd15b877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:42:33.924 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:33.923+0000 7fbd15b877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:42:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T22:42:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:42:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:42:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:42:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:42:34.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:34.551 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:42:34.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:42:34.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:34.552 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:42:34.918 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:34.915+0000 7fbd15b877c0 -1 Falling back to public interface 2024-09-15T22:42:35.099 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:36.101 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:42:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:42:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:42:36.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:37.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:37.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:37.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:42:37.547 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:42:37.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:37.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:42:37.967 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:38.060 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:38.059+0000 7fbd15b877c0 -1 osd.2 35 log_to_monitors true 2024-09-15T22:42:38.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:38.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:38.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:42:38.969 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:42:38.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:38.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:42:39.603 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 46 up_thru 19 down_at 36 last_clean_interval [19,35) [v2:127.0.0.1:6818/4157280410,v1:127.0.0.1:6819/4157280410] [v2:127.0.0.1:6820/4157280410,v1:127.0.0.1:6821/4157280410] exists,up 25c62bc0-9ca4-4b40-9539-7575bb06ae8f 2024-09-15T22:42:39.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:42:39.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:42:39.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:42:39.604 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-15T22:42:39.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:42:39.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:42:39.605 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:42:39.605 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:42:39.606 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:42:39.606 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:42:39.606 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:42:39.606 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:42:39.606 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:42:39.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:42:39.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:42:39.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:42:39.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:42:39.723 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:42:39.723 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:42:40.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:42:40.313 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:42:40.313 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:42:40.313 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:42:40.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:42:40.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:42:40.313 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:42:40.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803811 2024-09-15T22:42:40.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803811 2024-09-15T22:42:40.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811' 2024-09-15T22:42:40.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:42:40.466 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:42:40.703 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574880 2024-09-15T22:42:40.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574880 2024-09-15T22:42:40.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574880' 2024-09-15T22:42:40.704 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:42:40.704 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:42:40.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=197568495619 2024-09-15T22:42:40.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 197568495619 2024-09-15T22:42:40.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574880 2-197568495619' 2024-09-15T22:42:40.862 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:42:40.862 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:42:41.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117002 2024-09-15T22:42:41.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117002 2024-09-15T22:42:41.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574880 2-197568495619 3-115964117002' 2024-09-15T22:42:41.034 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:42:41.035 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:42:41.035 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803811 2024-09-15T22:42:41.036 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:42:41.037 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803811 2024-09-15T22:42:41.037 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:42:41.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803811 2024-09-15T22:42:41.039 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803811 2024-09-15T22:42:41.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-09-15T22:42:41.039 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:42:41.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803809 -lt 25769803811 2024-09-15T22:42:41.524 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:42:42.525 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:42:42.526 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:42:42.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803811 -lt 25769803811 2024-09-15T22:42:42.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:42:42.969 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:42:42.969 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574880 2024-09-15T22:42:42.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:42:42.971 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:42:42.971 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574880 2024-09-15T22:42:42.972 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574880 2024-09-15T22:42:42.972 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574880 2024-09-15T22:42:42.973 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574880' 2024-09-15T22:42:42.973 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:42:43.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574880 -lt 55834574880 2024-09-15T22:42:43.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:42:43.383 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-197568495619 2024-09-15T22:42:43.383 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:42:43.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:42:43.385 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-197568495619 2024-09-15T22:42:43.385 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:42:43.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=197568495619 2024-09-15T22:42:43.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2024-09-15T22:42:43.387 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 197568495619 2024-09-15T22:42:43.387 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:42:43.817 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 197568495619 -lt 197568495619 2024-09-15T22:42:43.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:42:43.818 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117002 2024-09-15T22:42:43.818 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:42:43.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:42:43.821 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117002 2024-09-15T22:42:43.821 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:42:43.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117002 2024-09-15T22:42:43.822 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117002' 2024-09-15T22:42:43.822 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 115964117002 2024-09-15T22:42:43.823 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:42:44.248 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117003 -lt 115964117002 2024-09-15T22:42:44.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:42:44.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:42:44.249 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:42:44.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:42:44.839 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:42:44.840 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:42:44.840 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:42:44.840 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:42:44.840 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:42:44.840 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:42:44.840 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:42:45.286 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:42:45.286 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:42:45.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:42:45.287 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:42:45.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:42:45.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:42:45.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:42:45.883 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_rados_get_with_subreadall_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:42:45.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:42:45.884 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:42:46.412 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:42:46.427 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:42:47.011 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:42:47.027 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:42:47.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:42:47.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:42:47.028 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:42:47.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:42:47.029 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:42:47.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:42:47.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:42:47.029 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:42:47.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:42:47.341 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:42:47.341 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:42:47.342 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:42:47.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:42:47.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:42:47.344 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:42:47.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:42:47.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:42:47.346 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:42:47.346 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:42:47.347 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:42:47.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:42:47.348 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:42:47.376 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:42:47.376 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:47.376 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:47.376 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:42:47.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:42:47.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:42:47.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T22:42:47.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T22:42:47.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-15T22:42:47.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-15T22:42:47.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:42:47.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:42:47.378 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:42:47.378 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:42:47.379 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:42:47.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:42:47.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:42:47.379 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:42:47.381 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:42:47.381 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:42:47.382 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:42:47.382 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:42:47.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:42:47.384 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:42:47.384 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:42:47.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:42:47.385 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:42:47.386 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:42:47.386 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:42:47.387 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:42:47.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:42:47.388 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:42:47.390 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:42:47.390 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:47.390 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:47.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:42:47.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:42:47.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:42:47.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-15T22:42:47.392 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:42:47.393 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:47.393 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:47.393 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.81867 2024-09-15T22:42:47.394 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:42:47.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:42:47.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:42:47.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T22:42:47.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T22:42:47.395 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-15T22:42:47.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:42:47.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:42:47.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:42:47.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-15T22:42:47.396 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T22:42:47.886 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:42:47.886 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:42:47.886 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:42:47.886 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:42:47.887 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:47.887 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:47.887 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:42:47.887 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T22:42:48.089 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:42:48.090 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:42:48.090 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:42:48.090 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:42:48.090 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:42:48.091 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:42:48.091 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:42:48.091 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:42:48.092 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:42:48.092 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:42:48.092 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:48.092 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:48.093 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:42:48.093 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:42:48.093 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get fsid 2024-09-15T22:42:48.213 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:42:48.213 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:42:48.213 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:42:48.213 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:42:48.213 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:42:48.214 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:42:48.214 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:42:48.214 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:42:48.215 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:42:48.215 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:48.215 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:48.215 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:42:48.215 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:42:48.216 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.81867/ceph-mon.a.asok config get mon_host 2024-09-15T22:42:48.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T22:42:48.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-15T22:42:48.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:42:48.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:42:48.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:42:48.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T22:42:48.327 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:42:49.246 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:42:49.246 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:42:49.246 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:42:49.247 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:42:49.247 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:49.247 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:49.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:42:49.248 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:42:49.249 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:42:49.310 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T22:42:49.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T22:42:49.600 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'rbd' already exists 2024-09-15T22:42:49.615 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:42:50.618 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T22:42:50.618 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:42:50.618 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:42:50.619 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:42:50.619 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:50.619 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:50.619 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-mon.a.asok 2024-09-15T22:42:50.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T22:42:50.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-mon.a.asok log flush 2024-09-15T22:42:50.720 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:42:50.732 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T22:42:50.735 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:42:47.927+0000 7f8c549fbe00 0 load: jerasure load: lrc 2024-09-15T22:42:50.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_1 td/test-erasure-eio 2024-09-15T22:42:50.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_rados_get_with_subreadall_eio_shard_1: local dir=td/test-erasure-eio 2024-09-15T22:42:50.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_rados_get_with_subreadall_eio_shard_1: local shard_id=1 2024-09-15T22:42:50.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_rados_get_with_subreadall_eio_shard_1: setup_osds 4 2024-09-15T22:42:50.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T22:42:50.736 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T22:42:50.737 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T22:42:50.739 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T22:42:50.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:42:50.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T22:42:50.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:42:50.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:42:50.743 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:42:50.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:42:50.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T22:42:50.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:42:50.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:42:50.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:42:50.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:42:50.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T22:42:50.744 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T22:42:50.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:42:50.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:42:50.745 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:42:50.745 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:42:50.745 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:42:50.745 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:42:50.745 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:42:50.745 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:42:50.745 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:42:50.746 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:42:50.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:42:50.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:42:50.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:42:50.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:42:50.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:42:50.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:42:50.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:42:50.746 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:42:50.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:42:50.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:42:50.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:42:50.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:42:50.747 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T22:42:50.748 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:42:50.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fae1cf5b-eeb0-4e36-a4c0-dee32b8181cc 2024-09-15T22:42:50.748 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 fae1cf5b-eeb0-4e36-a4c0-dee32b8181cc' 2024-09-15T22:42:50.748 INFO:tasks.workunit.client.0.smithi104.stdout:add osd0 fae1cf5b-eeb0-4e36-a4c0-dee32b8181cc 2024-09-15T22:42:50.749 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:42:50.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDqYudmYoSyLRAAb9S3Yii5Vmj3NOu2/UY0Xg== 2024-09-15T22:42:50.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDqYudmYoSyLRAAb9S3Yii5Vmj3NOu2/UY0Xg=="}' 2024-09-15T22:42:50.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fae1cf5b-eeb0-4e36-a4c0-dee32b8181cc -i td/test-erasure-eio/0/new.json 2024-09-15T22:42:51.060 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:42:51.075 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T22:42:51.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqYudmYoSyLRAAb9S3Yii5Vmj3NOu2/UY0Xg== --osd-uuid fae1cf5b-eeb0-4e36-a4c0-dee32b8181cc 2024-09-15T22:42:51.110 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:51.107+0000 7eff6f2ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:42:51.128 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:51.127+0000 7eff6f2ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:42:51.129 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:51.127+0000 7eff6f2ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:42:51.129 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:51.127+0000 7eff6f2ae7c0 -1 bdev(0x560456315c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:42:51.129 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:51.127+0000 7eff6f2ae7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T22:42:57.120 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T22:42:57.121 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:42:57.122 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd0 key to auth repository 2024-09-15T22:42:57.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:42:57.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:42:57.696 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.0 2024-09-15T22:42:57.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:42:57.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:42:57.697 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:42:57.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:42:57.698 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:42:57.729 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:57.727+0000 7f2e8a2217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:42:57.756 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:57.755+0000 7f2e8a2217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:42:57.757 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:57.755+0000 7f2e8a2217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:42:58.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:42:58.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:42:58.150 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:42:58.150 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:42:58.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:42:58.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:42:58.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:58.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:42:58.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:58.151 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:42:58.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:42:59.018 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:42:59.015+0000 7f2e8a2217c0 -1 Falling back to public interface 2024-09-15T22:42:59.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:42:59.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:42:59.622 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:42:59.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:42:59.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:42:59.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:00.122 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:01.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:01.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:01.124 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:43:01.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:43:01.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:01.125 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:01.567 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:01.864 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:43:01.863+0000 7f2e8a2217c0 -1 osd.0 0 log_to_monitors true 2024-09-15T22:43:02.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:02.569 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:02.570 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:43:02.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:43:02.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:02.570 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:03.173 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:04.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:04.175 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:04.175 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:43:04.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:43:04.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:04.176 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:04.688 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:05.690 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:05.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:05.691 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:43:05.691 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:43:05.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:05.692 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:06.133 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:07.135 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:07.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:07.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:43:07.136 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:43:07.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:07.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:07.711 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:08.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:08.713 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:08.713 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:43:08.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:43:08.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:08.714 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:09.153 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:10.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:10.154 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:10.154 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:43:10.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:43:10.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:10.155 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:10.600 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:11.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:11.602 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:11.602 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:43:11.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:43:11.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:11.603 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:12.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:13.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:13.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:13.039 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:43:13.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:43:13.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:13.040 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:13.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:14.728 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:14.729 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:43:14.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:14.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:43:14.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:14.729 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:15.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:16.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:16.928 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:43:16.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:16.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:43:16.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:16.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:17.519 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:18.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:18.521 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:18.521 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:43:18.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:43:18.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:18.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:19.621 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:20.622 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:20.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:20.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:43:20.623 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:43:20.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:20.623 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:21.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:22.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:22.363 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:22.364 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:43:22.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:43:22.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:22.364 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:23.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:24.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:24.041 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:24.041 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:43:24.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:43:24.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:24.042 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:24.635 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:25.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:25.638 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:25.638 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:43:25.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:43:25.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:25.639 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:43:26.110 INFO:tasks.workunit.client.0.smithi104.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1271611261,v1:127.0.0.1:6803/1271611261] [v2:127.0.0.1:6804/1271611261,v1:127.0.0.1:6805/1271611261] exists,up fae1cf5b-eeb0-4e36-a4c0-dee32b8181cc 2024-09-15T22:43:26.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:43:26.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:43:26.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:43:26.112 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T22:43:26.113 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:43:26.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:43:26.114 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:43:26.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:43:26.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:43:26.114 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:43:26.114 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:43:26.114 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:43:26.114 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:43:26.116 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:43:26.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:43:26.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:43:26.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:43:26.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:43:26.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:43:26.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:43:26.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:43:26.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:43:26.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:43:26.116 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:43:26.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:43:26.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:43:26.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T22:43:26.117 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:43:26.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=42074b0e-f4ac-4591-89da-584c34cb5dc8 2024-09-15T22:43:26.117 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 42074b0e-f4ac-4591-89da-584c34cb5dc8' 2024-09-15T22:43:26.117 INFO:tasks.workunit.client.0.smithi104.stdout:add osd1 42074b0e-f4ac-4591-89da-584c34cb5dc8 2024-09-15T22:43:26.117 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:43:26.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAOY+dm0VotCBAAa+wchIOJeIjbXIj7+1uabQ== 2024-09-15T22:43:26.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAOY+dm0VotCBAAa+wchIOJeIjbXIj7+1uabQ=="}' 2024-09-15T22:43:26.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 42074b0e-f4ac-4591-89da-584c34cb5dc8 -i td/test-erasure-eio/1/new.json 2024-09-15T22:43:26.677 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:43:26.693 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T22:43:26.695 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOY+dm0VotCBAAa+wchIOJeIjbXIj7+1uabQ== --osd-uuid 42074b0e-f4ac-4591-89da-584c34cb5dc8 2024-09-15T22:43:26.728 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:43:26.727+0000 7f57868797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:43:26.731 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:43:26.731+0000 7f57868797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:43:26.732 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:43:26.731+0000 7f57868797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:43:26.733 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:43:26.731+0000 7f57868797c0 -1 bdev(0x561941e7dc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:43:26.733 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:43:26.731+0000 7f57868797c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T22:43:33.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T22:43:33.328 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:43:33.329 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd1 key to auth repository 2024-09-15T22:43:33.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:43:33.330 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:43:33.915 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.1 2024-09-15T22:43:33.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:43:33.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:43:33.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:43:33.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:43:33.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:43:33.947 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:43:33.947+0000 7ff0f45c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:43:33.975 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:43:33.971+0000 7ff0f45c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:43:33.976 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:43:33.975+0000 7ff0f45c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:43:34.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:43:34.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:43:34.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:43:34.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:43:34.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:43:34.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:34.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:43:34.429 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:43:34.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:34.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:34.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:35.522 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:43:35.519+0000 7ff0f45c67c0 -1 Falling back to public interface 2024-09-15T22:43:35.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:35.915 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:35.916 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:43:35.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:43:35.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:35.916 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:36.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:37.398 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:37.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:37.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:43:37.399 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:43:37.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:37.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:37.818 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:38.532 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:43:38.531+0000 7ff0f45c67c0 -1 osd.1 0 log_to_monitors true 2024-09-15T22:43:38.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:38.820 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:38.820 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:43:38.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:43:38.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:38.821 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:39.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:40.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:40.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:40.344 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:43:40.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:43:40.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:40.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:40.960 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:41.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:41.961 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:41.962 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:43:41.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:43:41.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:41.962 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:42.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:43.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:43.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:43.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:43:43.469 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:43:43.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:43.470 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:43.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:44.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:44.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:44.903 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:43:44.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:43:44.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:44.904 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:45.397 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:46.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:46.399 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:46.400 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:43:46.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:43:46.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:46.400 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:46.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:47.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:47.924 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:47.925 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:43:47.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:43:47.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:47.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:48.350 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:49.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:49.371 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:49.372 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:43:49.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:43:49.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:49.373 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:49.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:50.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:50.900 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:50.901 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:43:50.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:43:50.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:50.901 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:51.643 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:52.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:52.645 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:52.645 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:43:52.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:43:52.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:52.646 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:53.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:54.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:54.463 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:54.464 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:43:54.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:43:54.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:54.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:54.963 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:55.964 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:55.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:55.965 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:43:55.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:43:55.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:55.965 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:57.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:43:58.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:43:58.276 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:43:58.277 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:43:58.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:43:58.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:43:58.277 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:43:59.134 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:00.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:00.136 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:00.136 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:44:00.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:44:00.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:00.137 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:44:00.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:01.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:01.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:01.938 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:44:01.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:44:01.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:01.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:44:02.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:03.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:03.515 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:03.515 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:44:03.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:44:03.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:03.516 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:44:03.949 INFO:tasks.workunit.client.0.smithi104.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1404547867,v1:127.0.0.1:6811/1404547867] [v2:127.0.0.1:6812/1404547867,v1:127.0.0.1:6813/1404547867] exists,up 42074b0e-f4ac-4591-89da-584c34cb5dc8 2024-09-15T22:44:03.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:44:03.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:44:03.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:44:03.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:44:03.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T22:44:03.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:44:03.951 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:44:03.952 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:44:03.953 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:44:03.953 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:44:03.953 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:44:03.953 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:44:03.953 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:44:03.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:44:03.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:44:03.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:44:03.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:44:03.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:44:03.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:44:03.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:44:03.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:44:03.954 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:44:03.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:44:03.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:44:03.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:44:03.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:44:03.955 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:44:03.955 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=12de6a95-12d2-42c7-a2c1-d9d40b9aa7a5 2024-09-15T22:44:03.955 INFO:tasks.workunit.client.0.smithi104.stdout:add osd2 12de6a95-12d2-42c7-a2c1-d9d40b9aa7a5 2024-09-15T22:44:03.956 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 12de6a95-12d2-42c7-a2c1-d9d40b9aa7a5' 2024-09-15T22:44:03.956 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:44:03.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAzY+dmIyA7OhAAbsKCWSTAaWCCw5F0v7qWzA== 2024-09-15T22:44:03.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAzY+dmIyA7OhAAbsKCWSTAaWCCw5F0v7qWzA=="}' 2024-09-15T22:44:03.978 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 12de6a95-12d2-42c7-a2c1-d9d40b9aa7a5 -i td/test-erasure-eio/2/new.json 2024-09-15T22:44:04.461 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:44:04.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T22:44:04.478 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzY+dmIyA7OhAAbsKCWSTAaWCCw5F0v7qWzA== --osd-uuid 12de6a95-12d2-42c7-a2c1-d9d40b9aa7a5 2024-09-15T22:44:04.512 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:04.510+0000 7fc097ed87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:04.515 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:04.514+0000 7fc097ed87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:04.517 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:04.514+0000 7fc097ed87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:04.517 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:04.514+0000 7fc097ed87c0 -1 bdev(0x559310e3bc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:44:04.517 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:04.514+0000 7fc097ed87c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T22:44:10.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T22:44:10.411 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:44:10.412 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd2 key to auth repository 2024-09-15T22:44:10.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:44:10.412 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:44:10.996 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:44:10.996 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:44:10.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:44:10.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:44:10.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:44:10.997 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:44:11.028 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:11.026+0000 7fe81c0807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:11.036 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:11.034+0000 7fe81c0807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:11.037 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:11.034+0000 7fe81c0807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:11.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:44:11.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:44:11.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:44:11.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:44:11.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:44:11.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:11.468 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:44:11.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:44:11.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:11.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:11.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:12.278 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:12.274+0000 7fe81c0807c0 -1 Falling back to public interface 2024-09-15T22:44:12.933 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:12.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:12.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:44:12.934 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:44:12.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:12.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:13.402 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:14.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:14.404 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:14.404 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:44:14.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:44:14.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:14.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:14.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:15.168 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:15.166+0000 7fe81c0807c0 -1 osd.2 0 log_to_monitors true 2024-09-15T22:44:15.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:15.834 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:15.835 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:44:15.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:44:15.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:15.835 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:16.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:17.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:17.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:17.467 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:44:17.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:44:17.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:17.467 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:18.035 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:19.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:19.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:19.037 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:44:19.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:44:19.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:19.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:19.485 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:20.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:20.487 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:20.487 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:44:20.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:44:20.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:20.488 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:20.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:21.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:21.949 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:21.949 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:44:21.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:44:21.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:21.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:22.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:23.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:23.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:23.445 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:44:23.445 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:44:23.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:23.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:23.866 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:24.868 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:24.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:24.869 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:44:24.869 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:44:24.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:24.870 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:25.407 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:26.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:26.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:26.409 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:44:26.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:44:26.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:26.409 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:27.093 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:28.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:28.095 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:28.095 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:44:28.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:44:28.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:28.096 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:28.830 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:29.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:29.832 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:29.833 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:44:29.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:44:29.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:29.833 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:30.358 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:31.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:31.360 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:31.361 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:44:31.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:44:31.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:31.361 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:32.432 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:33.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:33.434 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:33.435 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:44:33.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:44:33.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:33.435 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:34.351 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:35.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:35.353 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:35.353 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:44:35.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:44:35.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:35.354 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:36.209 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:37.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:37.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:37.212 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:44:37.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:44:37.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:37.213 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:38.337 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:39.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:39.339 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:39.339 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:44:39.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:44:39.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:39.340 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:39.808 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:40.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:40.810 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:40.810 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:44:40.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:44:40.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:40.811 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:44:41.313 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2499249758,v1:127.0.0.1:6819/2499249758] [v2:127.0.0.1:6820/2499249758,v1:127.0.0.1:6821/2499249758] exists,up 12de6a95-12d2-42c7-a2c1-d9d40b9aa7a5 2024-09-15T22:44:41.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:44:41.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:44:41.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:44:41.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T22:44:41.315 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:44:41.316 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:44:41.317 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:44:41.317 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:44:41.317 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:44:41.317 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:44:41.317 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:44:41.317 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:44:41.317 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:44:41.319 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:44:41.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:44:41.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:44:41.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:44:41.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:44:41.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:44:41.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:44:41.319 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:44:41.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:44:41.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:44:41.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:44:41.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:44:41.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:44:41.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T22:44:41.320 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:44:41.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=24a4776c-32f5-40a6-aeb4-ec3fd6d6c7e8 2024-09-15T22:44:41.320 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 24a4776c-32f5-40a6-aeb4-ec3fd6d6c7e8' 2024-09-15T22:44:41.320 INFO:tasks.workunit.client.0.smithi104.stdout:add osd3 24a4776c-32f5-40a6-aeb4-ec3fd6d6c7e8 2024-09-15T22:44:41.321 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:44:41.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBZY+dm+rNjFBAAbnM1kHPxIXKgVGZvwDfy2w== 2024-09-15T22:44:41.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBZY+dm+rNjFBAAbnM1kHPxIXKgVGZvwDfy2w=="}' 2024-09-15T22:44:41.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 24a4776c-32f5-40a6-aeb4-ec3fd6d6c7e8 -i td/test-erasure-eio/3/new.json 2024-09-15T22:44:41.982 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:44:42.000 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T22:44:42.001 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZY+dm+rNjFBAAbnM1kHPxIXKgVGZvwDfy2w== --osd-uuid 24a4776c-32f5-40a6-aeb4-ec3fd6d6c7e8 2024-09-15T22:44:42.035 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:42.034+0000 7f01257b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:42.045 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:42.042+0000 7f01257b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:42.047 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:42.042+0000 7f01257b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:42.047 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:42.042+0000 7f01257b67c0 -1 bdev(0x56318a0a5c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-15T22:44:42.047 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:42.046+0000 7f01257b67c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T22:44:48.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T22:44:48.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:44:48.450 INFO:tasks.workunit.client.0.smithi104.stdout:adding osd3 key to auth repository 2024-09-15T22:44:48.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T22:44:48.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:44:49.022 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.3 2024-09-15T22:44:49.022 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T22:44:49.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:44:49.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:44:49.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:44:49.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:44:49.054 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:49.050+0000 7fd557ce77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:49.066 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:49.062+0000 7fd557ce77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:49.068 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:49.066+0000 7fd557ce77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:44:49.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T22:44:49.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:44:49.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T22:44:49.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:44:49.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:44:49.446 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:49.447 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:44:49.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:44:49.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:49.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:44:49.857 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:50.546 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:50.542+0000 7fd557ce77c0 -1 Falling back to public interface 2024-09-15T22:44:50.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:50.858 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:50.859 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:44:50.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:44:50.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:50.860 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:44:51.311 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:52.312 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:52.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:52.313 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:44:52.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:44:52.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:52.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:44:52.725 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:53.310 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:44:53.306+0000 7fd557ce77c0 -1 osd.3 0 log_to_monitors true 2024-09-15T22:44:53.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:53.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:53.726 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:44:53.727 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:44:53.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:53.727 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:44:54.271 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:55.273 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:55.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:55.274 INFO:tasks.workunit.client.0.smithi104.stdout:4 2024-09-15T22:44:55.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:44:55.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:55.274 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:44:55.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:56.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:56.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:56.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:44:56.940 INFO:tasks.workunit.client.0.smithi104.stdout:5 2024-09-15T22:44:56.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:56.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:44:57.440 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:58.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:58.442 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:58.442 INFO:tasks.workunit.client.0.smithi104.stdout:6 2024-09-15T22:44:58.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:44:58.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:58.443 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:44:58.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:44:59.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:44:59.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:44:59.905 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:44:59.905 INFO:tasks.workunit.client.0.smithi104.stdout:7 2024-09-15T22:44:59.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:44:59.906 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:00.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:01.428 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:01.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:01.429 INFO:tasks.workunit.client.0.smithi104.stdout:8 2024-09-15T22:45:01.429 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:45:01.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:01.430 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:01.910 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:02.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:02.912 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:02.913 INFO:tasks.workunit.client.0.smithi104.stdout:9 2024-09-15T22:45:02.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:45:02.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:02.913 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:03.466 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:04.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:04.468 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:04.468 INFO:tasks.workunit.client.0.smithi104.stdout:10 2024-09-15T22:45:04.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:45:04.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:04.469 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:04.966 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:05.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:05.968 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:05.968 INFO:tasks.workunit.client.0.smithi104.stdout:11 2024-09-15T22:45:05.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:45:05.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:05.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:06.767 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:07.768 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:07.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:07.769 INFO:tasks.workunit.client.0.smithi104.stdout:12 2024-09-15T22:45:07.769 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:45:07.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:07.770 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:08.671 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:09.673 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:09.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:09.674 INFO:tasks.workunit.client.0.smithi104.stdout:13 2024-09-15T22:45:09.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:45:09.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:09.674 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:10.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:11.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:11.464 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:11.465 INFO:tasks.workunit.client.0.smithi104.stdout:14 2024-09-15T22:45:11.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:45:11.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:11.465 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:12.256 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:13.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:13.258 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:13.259 INFO:tasks.workunit.client.0.smithi104.stdout:15 2024-09-15T22:45:13.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:45:13.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:13.259 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:14.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:15.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:15.071 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:15.071 INFO:tasks.workunit.client.0.smithi104.stdout:16 2024-09-15T22:45:15.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:45:15.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:15.072 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:16.053 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:17.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:17.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:17.055 INFO:tasks.workunit.client.0.smithi104.stdout:17 2024-09-15T22:45:17.055 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:45:17.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:17.056 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:17.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:18.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:18.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:18.548 INFO:tasks.workunit.client.0.smithi104.stdout:18 2024-09-15T22:45:18.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:45:18.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:18.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T22:45:19.078 INFO:tasks.workunit.client.0.smithi104.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1208448292,v1:127.0.0.1:6827/1208448292] [v2:127.0.0.1:6828/1208448292,v1:127.0.0.1:6829/1208448292] exists,up 24a4776c-32f5-40a6-aeb4-ec3fd6d6c7e8 2024-09-15T22:45:19.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:45:19.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:45:19.078 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:45:19.079 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T22:45:19.079 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T22:45:19.079 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T22:45:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:45:19.080 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:45:19.081 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:45:19.081 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.0.asok 2024-09-15T22:45:19.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T22:45:19.081 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.0.asok log flush 2024-09-15T22:45:19.186 INFO:tasks.workunit.client.0.smithi104.stdout:{} 2024-09-15T22:45:19.198 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T22:45:19.204 INFO:tasks.workunit.client.0.smithi104.stdout:2024-09-15T22:42:59.027+0000 7f2e8a2217c0 0 load: jerasure load: lrc 2024-09-15T22:45:19.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_rados_get_with_subreadall_eio_shard_1: local poolname=pool-jerasure 2024-09-15T22:45:19.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:370: TEST_rados_get_with_subreadall_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T22:45:19.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:45:19.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T22:45:19.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T22:45:19.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T22:45:19.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T22:45:19.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T22:45:19.206 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T22:45:20.244 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T22:45:20.245 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T22:45:21.204 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' already exists 2024-09-15T22:45:21.226 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:45:22.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T22:45:22.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:45:22.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:45:22.228 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:45:22.229 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:45:22.230 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:45:22.230 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:45:22.231 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:45:22.231 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:45:22.231 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:45:22.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:45:22.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:45:22.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:45:22.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:45:22.362 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:45:22.363 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:45:22.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:45:22.925 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:45:22.925 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:45:22.925 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:45:22.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:45:22.925 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:45:22.925 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:45:23.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803808 2024-09-15T22:45:23.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803808 2024-09-15T22:45:23.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808' 2024-09-15T22:45:23.083 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:45:23.084 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:45:23.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574872 2024-09-15T22:45:23.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574872 2024-09-15T22:45:23.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808 1-55834574872' 2024-09-15T22:45:23.238 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:45:23.238 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:45:23.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378642 2024-09-15T22:45:23.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378642 2024-09-15T22:45:23.390 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808 1-55834574872 2-81604378642' 2024-09-15T22:45:23.391 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:45:23.391 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:45:23.542 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117003 2024-09-15T22:45:23.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117003 2024-09-15T22:45:23.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808 1-55834574872 2-81604378642 3-115964117003' 2024-09-15T22:45:23.543 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:45:23.543 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803808 2024-09-15T22:45:23.543 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:45:23.544 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:45:23.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803808 2024-09-15T22:45:23.545 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:45:23.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803808 2024-09-15T22:45:23.547 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803808 2024-09-15T22:45:23.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-09-15T22:45:23.547 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:45:24.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803806 -lt 25769803808 2024-09-15T22:45:24.023 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:45:25.025 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:45:25.026 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:45:25.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803808 -lt 25769803808 2024-09-15T22:45:25.444 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:45:25.445 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574872 2024-09-15T22:45:25.445 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:45:25.447 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:45:25.448 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574872 2024-09-15T22:45:25.448 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:45:25.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574872 2024-09-15T22:45:25.450 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574872 2024-09-15T22:45:25.450 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574872' 2024-09-15T22:45:25.450 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:45:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574872 -lt 55834574872 2024-09-15T22:45:25.944 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:45:25.945 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378642 2024-09-15T22:45:25.945 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:45:25.947 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:45:25.948 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378642 2024-09-15T22:45:25.948 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:45:25.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378642 2024-09-15T22:45:25.950 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 81604378642 2024-09-15T22:45:25.950 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378642' 2024-09-15T22:45:25.950 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:45:26.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378643 -lt 81604378642 2024-09-15T22:45:26.405 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:45:26.406 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117003 2024-09-15T22:45:26.406 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:45:26.408 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:45:26.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:45:26.409 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117003 2024-09-15T22:45:26.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117003 2024-09-15T22:45:26.410 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117003' 2024-09-15T22:45:26.410 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 115964117003 2024-09-15T22:45:26.411 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:45:26.897 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117004 -lt 115964117003 2024-09-15T22:45:26.897 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:45:26.898 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:45:26.898 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:45:27.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:45:27.477 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:45:27.477 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:45:27.478 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:45:27.478 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:45:27.478 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:45:27.478 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:45:27.478 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:45:27.903 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:45:27.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:45:27.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:45:27.904 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:45:28.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:45:28.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:45:28.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:45:28.437 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_rados_get_with_subreadall_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 recovery 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-81867-1 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-81867-1 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T22:45:28.438 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-81867-1 2024-09-15T22:45:28.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:45:28.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T22:45:28.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:45:28.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T22:45:28.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:45:28.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T22:45:28.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T22:45:28.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T22:45:28.439 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-81867-1 td/test-erasure-eio/ORIGINAL 2024-09-15T22:45:28.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81867-1 td/test-erasure-eio 1 2024-09-15T22:45:28.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-15T22:45:28.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T22:45:28.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T22:45:28.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T22:45:28.619 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-15T22:45:28.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T22:45:28.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-81867-1 2024-09-15T22:45:28.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T22:45:28.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-15T22:45:28.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T22:45:28.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-15T22:45:28.620 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T22:45:28.621 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-81867-1 2024-09-15T22:45:28.621 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:45:28.621 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-81867-1 2024-09-15T22:45:28.621 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-81867-1 2024-09-15T22:45:28.621 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:45:29.084 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:45:29.085 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:45:29.085 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:45:29.085 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:45:29.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T22:45:29.085 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T22:45:29.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-15T22:45:29.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-15T22:45:29.086 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-15T22:45:29.086 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T22:45:29.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T22:45:29.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T22:45:29.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T22:45:29.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T22:45:29.087 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T22:45:29.087 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T22:45:29.088 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T22:45:29.088 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T22:45:29.088 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T22:45:29.088 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:45:29.089 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:45:29.089 INFO:tasks.workunit.client.0.smithi104.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:45:29.089 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.1.asok 2024-09-15T22:45:29.089 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81867/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T22:45:29.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T22:45:29.210 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T22:45:29.211 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-15T22:45:29.211 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T22:45:29.211 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T22:45:29.211 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T22:45:29.212 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:45:29.212 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:45:29.212 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:45:29.212 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.81867/ceph-osd.1.asok 2024-09-15T22:45:29.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T22:45:29.212 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81867/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-81867-1 1 2024-09-15T22:45:29.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81867-1 2024-09-15T22:45:29.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T22:45:29.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T22:45:29.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81867-1 2024-09-15T22:45:29.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T22:45:29.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T22:45:29.332 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-81867-1 td/test-erasure-eio/COPY 2024-09-15T22:45:29.368 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T22:45:29.369 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T22:45:29.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-15T22:45:29.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-81867-1 2024-09-15T22:45:29.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:45:29.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-81867-1 2024-09-15T22:45:29.371 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-81867-1 2024-09-15T22:45:29.371 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:45:29.827 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:45:29.827 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:45:29.827 INFO:tasks.workunit.client.0.smithi104.stderr:2' 2024-09-15T22:45:29.827 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-15T22:45:29.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2024-09-15T22:45:29.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-15T22:45:29.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-15T22:45:29.827 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T22:45:29.828 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:45:29.828 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:45:29.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:45:29.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:45:29.828 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:45:30.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:45:30.138 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-15T22:45:30.916 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 is already out. 2024-09-15T22:45:30.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-81867-1 2024-09-15T22:45:30.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-15T22:45:30.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-15T22:45:30.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-81867-1 2024-09-15T22:45:30.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-81867-1 2024-09-15T22:45:30.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T22:45:31.461 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T22:45:31.461 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:45:31.461 INFO:tasks.workunit.client.0.smithi104.stderr:0' 2024-09-15T22:45:31.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 0 2024-09-15T22:45:31.462 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-15T22:45:32.105 INFO:tasks.workunit.client.0.smithi104.stderr:osd.2 is already in. 2024-09-15T22:45:32.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-15T22:45:32.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-15T22:45:32.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:45:32.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T22:45:32.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:45:32.123 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:45:32.124 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:45:32.126 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:45:32.126 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:45:32.126 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:45:32.126 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:45:32.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' 2024-09-15T22:45:32.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:45:32.126 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T22:45:32.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T22:45:32.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:45:32.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:45:32.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:45:32.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:45:32.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:45:32.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:45:32.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T22:45:32.127 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T22:45:32.127 INFO:tasks.workunit.client.0.smithi104.stdout:start osd.2 2024-09-15T22:45:32.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=52be3d08-a6d9-4775-a93c-cb6778fdc0be --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81867/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:45:32.128 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T22:45:32.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T22:45:32.128 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:45:32.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:45:32.129 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:45:32.163 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:45:32.157+0000 7f1e662517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:45:32.348 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:45:32.345+0000 7f1e662517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:45:32.349 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:45:32.345+0000 7f1e662517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:45:32.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T22:45:32.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:45:32.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:45:32.685 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:45:32.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:45:32.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:32.686 INFO:tasks.workunit.client.0.smithi104.stdout:0 2024-09-15T22:45:32.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:45:32.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:32.686 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:45:33.342 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:33.446 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:45:33.441+0000 7f1e662517c0 -1 Falling back to public interface 2024-09-15T22:45:34.343 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:34.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:34.344 INFO:tasks.workunit.client.0.smithi104.stdout:1 2024-09-15T22:45:34.344 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:45:34.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:34.345 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:45:34.788 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:35.791 INFO:tasks.workunit.client.0.smithi104.stdout:2 2024-09-15T22:45:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:45:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:45:36.237 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:45:36.744 INFO:tasks.workunit.client.0.smithi104.stderr:2024-09-15T22:45:36.741+0000 7f1e662517c0 -1 osd.2 35 log_to_monitors true 2024-09-15T22:45:37.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:45:37.239 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:45:37.239 INFO:tasks.workunit.client.0.smithi104.stdout:3 2024-09-15T22:45:37.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:45:37.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:45:37.240 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:45:37.796 INFO:tasks.workunit.client.0.smithi104.stdout:osd.2 up in weight 1 up_from 46 up_thru 19 down_at 36 last_clean_interval [19,35) [v2:127.0.0.1:6818/2416662578,v1:127.0.0.1:6819/2416662578] [v2:127.0.0.1:6820/2416662578,v1:127.0.0.1:6821/2416662578] exists,up 12de6a95-12d2-42c7-a2c1-d9d40b9aa7a5 2024-09-15T22:45:37.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:45:37.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:45:37.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:45:37.797 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-15T22:45:37.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:45:37.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:45:37.798 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:45:37.798 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:45:37.798 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:45:37.799 INFO:tasks.workunit.client.0.smithi104.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:45:37.799 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:45:37.799 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:45:37.799 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:45:37.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:45:37.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:45:37.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:45:37.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:45:37.909 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:45:37.910 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:45:38.593 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:45:38.594 INFO:tasks.workunit.client.0.smithi104.stderr:1 2024-09-15T22:45:38.594 INFO:tasks.workunit.client.0.smithi104.stderr:2 2024-09-15T22:45:38.594 INFO:tasks.workunit.client.0.smithi104.stderr:3' 2024-09-15T22:45:38.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:45:38.594 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:45:38.595 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:45:38.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803813 2024-09-15T22:45:38.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803813 2024-09-15T22:45:38.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803813' 2024-09-15T22:45:38.922 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:45:38.922 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:45:39.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574877 2024-09-15T22:45:39.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574877 2024-09-15T22:45:39.076 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803813 1-55834574877' 2024-09-15T22:45:39.077 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:45:39.077 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:45:39.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=197568495619 2024-09-15T22:45:39.313 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 197568495619 2024-09-15T22:45:39.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803813 1-55834574877 2-197568495619' 2024-09-15T22:45:39.314 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:45:39.314 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T22:45:39.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117009 2024-09-15T22:45:39.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117009 2024-09-15T22:45:39.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803813 1-55834574877 2-197568495619 3-115964117009' 2024-09-15T22:45:39.548 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:45:39.548 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803813 2024-09-15T22:45:39.548 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:45:39.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:45:39.550 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803813 2024-09-15T22:45:39.550 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:45:39.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803813 2024-09-15T22:45:39.551 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-09-15T22:45:39.551 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.0 seq 25769803813 2024-09-15T22:45:39.551 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:45:40.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803811 -lt 25769803813 2024-09-15T22:45:40.067 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:45:41.069 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:45:41.070 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:45:41.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803813 -lt 25769803813 2024-09-15T22:45:41.509 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:45:41.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574877 2024-09-15T22:45:41.510 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:45:41.511 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:45:41.512 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574877 2024-09-15T22:45:41.512 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:45:41.513 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574877 2024-09-15T22:45:41.514 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.1 seq 55834574877 2024-09-15T22:45:41.514 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574877' 2024-09-15T22:45:41.514 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:45:41.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574877 -lt 55834574877 2024-09-15T22:45:41.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:45:41.932 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-197568495619 2024-09-15T22:45:41.932 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:45:41.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:45:41.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-197568495619 2024-09-15T22:45:41.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:45:41.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=197568495619 2024-09-15T22:45:41.937 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.2 seq 197568495619 2024-09-15T22:45:41.937 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2024-09-15T22:45:41.937 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:45:42.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 197568495619 -lt 197568495619 2024-09-15T22:45:42.370 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:45:42.370 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117009 2024-09-15T22:45:42.371 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:45:42.372 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T22:45:42.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117009 2024-09-15T22:45:42.373 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:45:42.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117009 2024-09-15T22:45:42.374 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117009' 2024-09-15T22:45:42.375 INFO:tasks.workunit.client.0.smithi104.stdout:waiting osd.3 seq 115964117009 2024-09-15T22:45:42.375 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T22:45:42.881 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117009 -lt 115964117009 2024-09-15T22:45:42.881 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:45:42.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:45:42.882 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:45:43.490 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T22:45:43.491 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:45:43.491 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:45:43.491 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:45:43.491 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:45:43.491 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:45:43.491 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:45:43.491 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:45:43.889 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T22:45:43.890 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:45:43.890 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:45:43.891 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:45:44.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-15T22:45:44.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-15T22:45:44.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T22:45:44.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-15T22:45:44.448 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T22:45:44.449 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T22:45:44.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T22:45:44.549 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:45:44.550 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:45:44.550 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:45:44.550 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:45:44.551 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:45:44.551 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:45:44.551 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:45:44.969 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T22:45:44.970 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:45:44.970 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:45:44.970 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:45:45.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T22:45:45.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:45:45.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:45:45.522 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:374: TEST_rados_get_with_subreadall_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-15T22:45:45.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T22:45:45.523 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T22:45:46.027 INFO:tasks.workunit.client.0.smithi104.stderr:pool 'pool-jerasure' does not exist 2024-09-15T22:45:46.043 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T22:45:46.561 INFO:tasks.workunit.client.0.smithi104.stderr:erasure-code-profile myprofile does not exist 2024-09-15T22:45:46.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T22:45:46.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:45:46.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:45:46.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:45:46.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:45:46.578 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:45:46.578 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:45:46.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:45:46.579 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:45:46.890 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:45:46.890 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:45:46.891 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:45:46.892 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:45:46.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:45:46.893 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:45:46.894 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:45:46.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:45:46.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:45:46.896 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:45:46.897 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:45:46.898 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:45:46.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:45:46.899 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:45:46.926 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:45:46.926 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:45:46.926 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:45:46.926 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:45:46.927 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:45:46.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:45:46.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T22:45:46.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/test-erasure-eio 0 2024-09-15T22:45:46.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-15T22:45:46.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T22:45:46.928 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T22:45:46.929 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:45:46.929 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:45:46.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:45:46.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:45:46.929 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:45:46.931 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:45:46.931 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:45:46.932 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:45:46.933 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:45:46.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:45:46.934 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:45:46.935 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:45:46.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:45:46.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:45:46.936 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:45:46.936 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:45:46.938 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:45:46.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T22:45:46.939 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-15T22:45:46.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:45:46.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:45:46.940 INFO:tasks.workunit.client.0.smithi104.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.81867 2024-09-15T22:45:46.940 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.81867 2024-09-15T22:45:46.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:45:46.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:45:46.942 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T22:45:46.943 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T22:45:46.943 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T22:45:46.995 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2024-09-15T22:45:46.996 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T22:45:47.366 DEBUG:teuthology.parallel:result is None 2024-09-15T22:45:47.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T22:45:47.377 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T22:45:47.377 DEBUG:teuthology.orchestra.run.smithi104:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T22:45:47.440 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T22:45:47.440 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T22:45:47.455 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T22:45:47.455 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T22:45:47.520 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T22:45:47.529 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T22:45:47.529 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T22:45:47.553 INFO:teuthology.orchestra.run.smithi104.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T22:45:47.553 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================== 2024-09-15T22:45:47.553 INFO:teuthology.orchestra.run.smithi104.stdout:+hv01.front.sepi 67.205.162.81 3 u 96 128 377 0.181 -1.771 0.074 2024-09-15T22:45:47.553 INFO:teuthology.orchestra.run.smithi104.stdout:+hv02.front.sepi 96.245.170.99 3 u 45 128 377 0.158 +0.158 0.158 2024-09-15T22:45:47.554 INFO:teuthology.orchestra.run.smithi104.stdout:*hv03.front.sepi 74.6.168.72 3 u 3 128 377 0.152 -0.421 0.284 2024-09-15T22:45:47.554 INFO:teuthology.orchestra.run.smithi104.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-15T22:45:47.554 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T22:45:47.568 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T22:45:47.569 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T22:45:47.599 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T22:45:47.671 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T22:45:47.703 INFO:teuthology.task.internal:Duration was 5394.334391 seconds 2024-09-15T22:45:47.703 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T22:45:47.736 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T22:45:47.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T22:45:47.774 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T22:45:47.774 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-09-15T22:45:47.774 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T22:45:47.825 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T22:45:47.825 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T22:45:47.907 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T22:45:47.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T22:45:48.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T22:45:48.016 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T22:45:48.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T22:45:48.027 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T22:45:48.081 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:45:48.091 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-09-15T22:45:48.101 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:45:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:45:48.147 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T22:45:48.159 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T22:45:48.164 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905692/remote/smithi104 2024-09-15T22:45:48.166 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T22:45:48.566 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T22:45:48.566 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T22:45:48.615 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T22:45:48.674 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T22:45:48.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T22:45:48.741 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T22:45:48.741 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T22:45:48.748 INFO:teuthology.orchestra.run.smithi104.stdout: 658614 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 22:45 /home/ubuntu/cephtest 2024-09-15T22:45:48.749 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T22:45:48.866 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T22:45:48.933 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} duration: 5394.334391117096 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-15T22:45:48.933 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T22:45:49.024 INFO:teuthology.run:pass